Airflow Middleware Onboarding

The ingest project requires the use of Airflow as a Middleware layer to be running in AKS so that Ingest Services can leverage Airflow as a Workflow Engine.

  • Architecture Design of Required Azure Resources Necessary for Airflow
  1. Postgres
  2. Redis
  3. File Storage
  • Host 3rd Party Source Code
  1. airflow-function
  2. airflow-statsd
  • GitLab Pipeline required to containerize and host containers
  1. airflow-function
  2. airflow-statsd
  • Host Helm Charts for installation
  1. osdu-airflow

Automation Onboarding

  • create Pipelines for airflow deployment

  • Update helm template task to run python script to add namespace for generated airflow yamls

  • Update git ops task to copy the charts generated from airflow.targz in different folder to flux repository

  • Execute Installation in Terrforom

  1. osdu-airflow

Acceptance Criteria

  1. Airflow Installs automatically as part of the service_resources template.
  2. All Tests Pass
  3. All Pipelines Pass
  4. Documentation Exists
  5. Services are able to leverage Airflow Workflow Engine
Edited by Daniel Scholl (MS]