Develop manifest changes to support ingestion of master/wp/wpc/dataset/file, etc. [AWS supplier => Azure consumer]
As an EDS Solution Architect, I need the EDS Fetch and Ingest service to leverage the OSDU standard manifest ingestion endpoint in development by the ingestion team so that we can have a consistent way to ingest data.
Implications: The Fetch and Ingest python code will need to modified to convert the incoming data (fetched data) into a chained (related) manifest that complies with the Data Definition manifest schema. This manifest will need to be submitted to the ingestion endpoint.
Acceptance Criteria
- Any group type (master data, work product, work product component, dataset, file, etc.) data is converted to a manifest
- The only ingestion that occurs happens through manifest ingestion
- Unit tests pass
- Merge to master is complete
Tasks
-
Understand the manifest schema from Data Definitions -
Create unit tests that reflect desired outcomes -
Modify python code to format incoming data into validated manifest(s) and pass to ingestion endpoint -
Create merge requests