ADR: Move Ingestion Logic from Python SDK to the separate package.
The main goal of Python SDK
was to interface with OSDU microservices' API.
It'd been so, until we moved Manifest Based Ingestion common code from Ingestion DAG
to it. And now Python SDK
is responsible not only for interacting with OSDU microservices, but for implementing Manifest Based Ingestion.
We believe that such mixing of two different logics is not proper. Imagine, we want to use Python SDK only for communicating with Entitlements service (or other Service), so we don't need Ingestion logic there at all.
So, it was decided to move Manifest Ingestion logic into separate package that is only responsible for Ingestion process. This new package is cloud-agnostic and it is used mainly in Airflow operators.
The new package's repository is https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib.
There are a few MRs that implement using this new package. OSDU Ingestion Lib: osdu/platform/data-flow/ingestion/osdu-ingestion-lib!1 (merged)
Python SDK: !47 (merged)
OSDU Airflow Lib: osdu/platform/data-flow/ingestion/osdu-airflow-lib!6 (merged)