Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • D Data Workflow
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 1
    • Issues 1
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Data Flow
  • Data IngestionData Ingestion
  • Data Workflow
  • Issues
  • #1
Closed
Open
Issue created Jun 16, 2021 by Kateryna Kurach (EPAM)@Kateryna_KurachDeveloper

Workflow services updates to be compatible with Airflow 2.0 - Common Code

Scope

Workflow Service updates have to be done to be compatible with Airflow 2.0. It also should be backward compatible with Airflow 1.0.

Decision

Currently, Workflow service triggers a DAG by direct Airflow experimental API call.

Requirements:

  • Implement logic to call Aifrlow 2.0 stable API
  • Make it optional to use one of Airflow API versions
  • Provide interface for support of separate CPS logic (the method can be overridden by CSP)

Description:

In workflow-core module should be implemented functionality to communicate with Airflow 2.0 , which will be conditional on property, with provider specific authorization service.

For instance, GCP deploys Airflow DAGs on Google managed Composer service. It brings an additional level of authentication, therefore it will have different logic to trigger Airflow 2.0 DAG within Workflow Service. Regarding to the current implementation it is impossible to support an additional auth logic.

That is why the solution implies of the core logic implementation to support both Airflow API versions and provide separate implementation for each CSP.

Rationale

Migration to Airflow 2.0 for all CSPs could be long running process, we need to let the CSPs decide when they can migrate.

Optional (could be done in separate issue)

It can be separate from this ADR, but with new stable API https://airflow.apache.org/docs/apache-airflow/2.0.0/stable-rest-api-ref.html#operation/get_dag_run Workflow service can ask DAG run status directly from Airflow, instead of keeping it in DB and relying on DAGs update their state manually.

There are currently no suggestions for changing the workflow endpoints, but they may come later.

Edited Aug 13, 2021 by Siarhei Khaletski (EPAM)
Assignee
Assign to
Time tracking