Custom Operator Metadata - Infrastructure Requirement (Cosmos Collection)
We need to support creation of custom operators to enhance composability in ingestion workflows. To support this we need to store the metadata associated to each custom operator. We need a new cosmos collection which can store this metadata information, we need create this per data partition
Collection name: WorkflowCustomOperatorInfo
Collection Properties: Partition key on custom operator name
Why is this change needed?
To Store the metadata information for every custom operator added into ingestion
No such collection available to store information related to custom operator metadata
A new cosmos collection is created and we will be able to store the metadata related to custom opearators in this cosmos collection.