Airflow Middleware Onboarding

The ingest project requires the use of Airflow as a Middleware layer to be running in AKS so that Ingest Services can leverage Airflow as a Workflow Engine.

• Architecture Design of Required Azure Resources Necessary for Airflow

1. Postgres
2. Redis
3. File Storage

• Host 3rd Party Source Code

1. airflow-function
2. airflow-statsd

• GitLab Pipeline required to containerize and host containers

1. airflow-function
2. airflow-statsd

• Host Helm Charts for installation

1. osdu-airflow

Automation Onboarding

• create Pipelines for airflow deployment

• Update helm template task to run python script to add namespace for generated airflow yamls

• Update git ops task to copy the charts generated from airflow.targz in different folder to flux repository

• Execute Installation in Terrforom

1. osdu-airflow

---

Acceptance Criteria

1. Airflow Installs automatically as part of the service_resources template.
2. All Tests Pass
3. All Pipelines Pass
4. Documentation Exists
5. Services are able to leverage Airflow Workflow Engine