Skip to content

POC changes for Manifest By Reference

Priyanka Bhongade requested to merge feature-m19-manifest-by-reference-priyanka into master
  • POC change in airflow utility for fetching dataset URL from airflow
  • Added HTTP REQUEST Constant , SIZE Constant ,PAYLOAD LIMIT Constant and FILE.GENERIC DATASET Constant
  • Added Json Frameup for Registering dataset
  • Added Json Frameup for uploading records file
  • Added Json Frameup for triggering Manifest by Reference ingestion
  • Added file manifest_by_reference_processing.py with functionalities of getting storageInstruction, uploading records json file and Registering dataset
  • POC/Logic updation in Main src_dags_fetch_and_ingest file to trigger either Osdu_ingest or Osdu_ingest_by_reference
  • Cleaned static values from workflow Frameup

Merge request reports