Skip to content

Enable Support for Packaged DAGs

The ADR for package DAG support has been approved and we need to structure the Ingestion Dags repository to support packaged DAGs. The new structure will look like as below

New folder structure

├── osdu_manifest
│   ├── __init__.py
│   ├── libs
│   │   ├── __init__.py
│   │   └── utils.py
│   └── operators
│   |    ├── __init__.py
│   |    └── customOperator1.py
|   |___ hooks
|   |    |__  __init__.py
|   | 
|   |___ configs
|        |__ __init__.py
|
|___ osdu-ingest-r3.py

Changes to support this will include

  • Restructuring the folders
  • Fixing any import statements
  • Minor changes to run existing tests