Airflow Middleware Onboarding
The ingest project requires the use of Airflow as a Middleware layer to be running in AKS so that Ingest Services can leverage Airflow as a Workflow Engine.
Architecture Design of Required Azure Resources Necessary for Airflow
- File Storage
Host 3rd Party Source Code
GitLab Pipeline required to containerize and host containers
Host Helm Charts for installation
create Pipelines for airflow deployment
Update helm template task to run python script to add namespace for generated airflow yamls
Update git ops task to copy the charts generated from airflow.targz in different folder to flux repository
Execute Installation in Terrforom
- Airflow Installs automatically as part of the service_resources template.
- All Tests Pass
- All Pipelines Pass
- Documentation Exists
- Services are able to leverage Airflow Workflow Engine