Airflow Middleware Onboarding
The ingest project requires the use of Airflow as a Middleware layer to be running in AKS so that Ingest Services can leverage Airflow as a Workflow Engine.
-
Architecture Design of Required Azure Resources Necessary for Airflow
- Postgres
- Redis
- File Storage
-
Host 3rd Party Source Code
- airflow-function
- airflow-statsd
-
GitLab Pipeline required to containerize and host containers
- airflow-function
- airflow-statsd
-
Host Helm Charts for installation
- osdu-airflow
Automation Onboarding
-
create Pipelines for airflow deployment -
Update helm template task to run python script to add namespace for generated airflow yamls -
Update git ops task to copy the charts generated from airflow.targz in different folder to flux repository -
Execute Installation in Terrforom
- osdu-airflow
Acceptance Criteria
- Airflow Installs automatically as part of the service_resources template.
- All Tests Pass
- All Pipelines Pass
- Documentation Exists
- Services are able to leverage Airflow Workflow Engine