Workflow services updates to be compatible with Airflow 2.0 - Common Code
Scope
Workflow Service updates have to be done to be compatible with Airflow 2.0. It also should be backward compatible with Airflow 1.0.
Decision
Currently, Workflow service triggers a DAG by direct Airflow experimental API call.
Requirements:
- Implement logic to call Aifrlow 2.0 stable API
- Make it optional to use one of Airflow API versions
- Provide interface for support of separate CPS logic (the method can be overridden by CSP)
Description:
In workflow-core module should be implemented functionality to communicate with Airflow 2.0 , which will be conditional on property, with provider specific authorization service.
For instance, GCP deploys Airflow DAGs on Google managed Composer service. It brings an additional level of authentication, therefore it will have different logic to trigger Airflow 2.0 DAG within Workflow Service. Regarding to the current implementation it is impossible to support an additional auth logic.
That is why the solution implies of the core logic implementation to support both Airflow API versions and provide separate implementation for each CSP.
Rationale
Migration to Airflow 2.0 for all CSPs could be long running process, we need to let the CSPs decide when they can migrate.
Optional (could be done in separate issue)
It can be separate from this ADR, but with new stable API https://airflow.apache.org/docs/apache-airflow/2.0.0/stable-rest-api-ref.html#operation/get_dag_run Workflow service can ask DAG run status directly from Airflow, instead of keeping it in DB and relying on DAGs update their state manually.
There are currently no suggestions for changing the workflow endpoints, but they may come later.