README.md



To learn more about this project, read the wiki.


Ingestion DAGs

Contents

Introduction
Deployment


GCP Composer


Installing Python Dependencies


DAG Implementation Details
Required Variables


Internal Services


Configuration


Ingestion


OSDU Python SDK


Testing


Running E2E Tests


Logging


Logging Configuration


Remote Logger


DAGs Description


Osdu_ingest


Osdu_ingest R2


Operators Description


Workflow Status Operator


Validate Manifest Schema Operator


Ensure Manifest Integrity Operator


Process Manifest Operator


Backward compatibility
Licence


Introduction
The project is a set of Apache Airflow DAGs implementations to orchestrate data ingestion within OSDU platform.
The following DAGs are implemented:

Osdu_ingest - R3 Manifest Ingestion DAG
Osdu_ingest_r2 - R2 Manifest Ingestion DAG (deprecated)


Deployment

GCP Composer
GCP provides Cloud Composer a fully managed workflow orchestration service built on Apache Airflow.
To deploy the Ingestion DAGs on GCP Cloud Composer just upload files from /src folder into DAGS_FOLDER and PLUGINS_FOLDER accordingly into the DAG bucket that provided by Composer environment. More info in documentation.