The OSDU R2 Prototype includes a Workflow Engine, an implementation of Apache Airflow, to orchestrate business
processes. In particular, the Workflow Engine handles ingestion of opaque and well log .las files in OSDU R2.
The Workflow Engine encompasses the following components:
* Opaque Ingestion DAG
* OSDU Ingestion DAG
* Workflow Status Operator
* Stale Jobs Scheduler
* Workflow Finished Sensor Operator
The project is a set of Apache Airflow DAGs implementations to orchestrate data ingestion within OSDU platform.
The following DAGs are implemented:
* Osdu_ingest - R3 Manifest Ingestion DAG
* Osdu_ingest_r2 - R2 Manifest Ingestion DAG
## Deployment
...
...
@@ -48,15 +42,14 @@ GCP provides Cloud Composer a fully managed workflow orchestration service built
To deploy the Ingestion DAGs on GCP Cloud Composer just upload files from */src* folder into *DAGS_FOLDER* and *PLUGINS_FOLDER* accordingly into the DAG bucket that provided by Composer environment. [More info in documentation.](https://cloud.google.com/composer/docs/quickstart#uploading_the_dag_to)
*DAGS_FOLDER* and *FLUGINS_FOLDER* are setting up in airflow.cfg file.
*DAGS_FOLDER* and *PLUGINS_FOLDER* are setting up by Composer itself.
According to the [DAG implementation details](#dag-implementation-details) need to put [osdu_api] directory into the *DAGS_FOLDER*. Moreover, all required variables have to be set in Airflow meta store by Variables mechanism. [List of the required variables](#required-variables).
### Installing Python Dependencies
#### Installing Python Dependencies
Environment dependencies might be installed by several ways:
1. Installing a Python dependency from PyPI. Cloud Composer picks up *requirements.txt* file from the DAGs bucket.
2. Setting up an environment into the Cloud Composer Console.
3. Installing local Python library. Put your dependencies into *DAG_FOLDER/libs* directory. Airflow automatically adds *DAG_FOLDER* and *PLUGINS_FOLDER* to the *PATH*.
1. Setting up an environment into the Cloud Composer Console.
2. Installing local Python library. Put your dependencies into *DAG_FOLDER/libs* directory. Airflow automatically adds *DAG_FOLDER* and *PLUGINS_FOLDER* to the *PATH*.
