Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • I Ingestion DAGs
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 32
    • Issues 32
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1
    • Merge requests 1
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Data Flow
  • Data IngestionData Ingestion
  • Ingestion DAGs
  • Issues
  • #71
Closed
Open
Issue created Jun 30, 2021 by Ben Lasscock@blasscoc💬Maintainer17 of 68 checklist items completed17/68 checklist items

Airflow 2.0 WIP

Tracking the task list for the Airflow 2.0 port described here.

Install Airflow 2.0 to all environments

Azure

  • Not Started
  • In Progress
  • Blocking
  • Completed

AWS

  • Not Started
  • In Progress
  • Blocking
  • Completed

GCP

  • Not Started
  • In Progress
  • Blocking
  • Completed

IBM issue M9

  • Not Started
  • In Progress
  • Blocking
  • Completed

Airflow 2.0 required DAG code changes (level of effort estimated at 5-days)

The strategy is for (EPAM) to deliver code changes for the manifest based ingestion DAGs issue and the WITSML parser issue which will demonstrate the required changes to the Python code to support both Airflow 2.+ and a deprecating strategy to continue support for Airflow 1.10.X.

2.1 Manifest-based ingestion (Python) [issue] @Kateryna_Kurach (#60 (closed))

  • Not Started
  • In Progress - Planned delivery tagged for M7
  • Blocking
  • Completed

2.2 WITSML parser (Python) issue @Kateryna_Kurach

  • Not Started
  • In Progress - Planned delivery tagged for M7
  • Blocking
  • Completed

2.3 SEGY -> OpenVDS - Planned delivery tagged for M8

  • Not Started
  • In Progress
  • Blocking
  • Completed

SEGY -> ZGY issue @sacha

  • Not Started
  • In Progress
  • Blocking
  • Completed

CSV Parser issue @tdixon @sdubey7 @frubio (M8 by EPAM)

  • Not Started
  • In Progress
  • Blocking
  • Completed

Workflow Services (level of effort estimated at 10-days)

Plans for updating the workflow services to be compatible with Airflow 2.0 is detailed here issue. (EPAM) has taken on this work.

Implement Aifrlow 2.0 stable API in workflow-core, make it optional.

  • Not Started - Planned delivery tagged for M8 M9
  • In Progress
  • Blocking
  • Completed

Azure Migration @kibattul delivery M9

  • Not Started
  • In Progress
  • Blocking
  • Completed

AWS Migration @Wibben

  • Not Started
  • In Progress
  • Blocking
  • Completed

GCP Migration @Kateryna_Kurach M9

  • Not Started
  • In Progress
  • Blocking
  • Completed

IBM Migration @shrikgar M8

  • Not Started
  • In Progress
  • Blocking
  • Completed

4. Testing and pre-shipping

The dev team (??) will need to provide clear instructions for the Airflow upgrade which the CSP can follow in both QA and pre-ship environments so that it is prepped for testing by respective teams.

Create Airflow upgrade instructions.

  • Not Started
  • In Progress
  • Blocking
  • Completed

QA - Airflow 2.+ readiness

  • Not Started
  • In Progress
  • Blocking
  • Completed

Pre-shipping - Airflow 2.+ readiness

  • Not Started
  • In Progress
  • Blocking
  • Completed
Edited Mar 09, 2022 by Chris Zhang
Assignee
Assign to
Time tracking