Manifest Ingestion: Refactor to have standalone capability
In seeking maximum reusability, the manifest ingestion workflow should take a bottoms-up approach where:
- Each DAG operator is Airflow agnostic and capable of running within a Python runtime environment without Airflow
- Each DAG operator is able to run as a script taking the appropriate inputs, performing its work, and then providing the expected outputs (interacting with OSDU services is expected)
- A DAG workflow should run end-to-end without requiring Airflow - in essence, running as a script from the command line with the correct inputs
- The DAG workflow should encapsulate the above into an Airflow workflow
The workflow should be executable outside of Airflow. The Airflow components should abstract the Airflow pieces from the core workflow itself.