Move Airflow common logic to osdu-airflow-lib project
Some of DAGs have dependencies on code from Ingestion DAGs repository. For instance, the MRs bring updates for parsers DAGs:
- segy-to-vds-conversion!24 (merged)
- segy-to-zgy-conversion!36 (merged)
- osdu/platform/data-flow/ingestion/csv-parser/csv-parser!149 (merged)
this requires UpdateStatusOperator
class from Ingestion DAGs project. It means, that it will require to deploy Ingestion DAGs code into environment to be able to use it for DAGs from MRs above.
The real case now is for WISTML Parser, where we have to add osdu_manifest
code to use operators for WITSML Parser DAG steps.
Expects: All the Airflow related logic (operators, hooks, etc.) is able to be installed into environment independently (using pip) of Ingestion DAGs code base.