Need common DAG deployment framework
At present, we lack a standard framework for deploying DAGs. There are multiple perspectives to consider:
- The deployment of code, which must also consider dependency conflicts
- The registering of DAGs. Presently, this is facilitated via the latest Workflow APIs
Note some approaches in use today:
- CI/CD pipelines used to deploy code for security purposes
- IBM using Git Sync process for deploying DAGs
Considerations:
- Standardizing DAG Operator deployment with things like the Kubernetes Pod Operator