Design CI/CD strategy for Airflow DAGs
As a Software Engineer, I want a consistent approach for CI/CD pipelines to deliver DAGs (directed acyclic graphs, or more simply python code modules) to a Cloud Service Provider's (CSP) Airflow installation so that EDS deliverables can be reliably deployed to cloud provider environments in automated fashion.
Acceptance Criteria
- Other teams that may already have CI/CD pipelines deploying DAGs are consulted so their experience can be leveraged
- Design approach is documented and vetted with at least two CSPs
- Versions of the technology stack (Python, Airflow, etc.) are documented
- CI/CD pipeline meets OSDU standards and requirements.
- EDS wiki is updated
Tasks
-
Identify teams that have existing DAG CI/CD pipelines and engage them for knowledge sharing -
Design approach -
Vet with at least two cloud providers -
Update wiki -
Share with EDS CSPs