Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • H Home
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 15
    • Issues 15
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Data Flow
  • Data IngestionData Ingestion
  • Home
  • Issues
  • #40
Closed
Open
Issue created Sep 24, 2020 by Stephen Whitley (Invited Expert)@stephenwhitley2 of 6 checklist items completed2/6 checklist items

Apache AirFlow for ingestion

Using Apache Airflow to support Workflow Orchestration for Ingestion Workflows

Status

  • Initiated
  • Proposed
  • Trialing
  • Under review
  • Approved
  • Retired

Decision

We will use Apache Airflow for implementing and executing ingestion workflows. The will leverage the technology as an orchestration system and AirFlow Operators can be both built in, or developed as custom operators.

Rationale

We need a cross platform workflow orchestration so that we can reuse both ingestion workflows and operators (tasks) in the workflows. This will be an area of high reuse across OSDU members.

Consequences

This is an Open Source solution that will have to be managed within the OSDU Platform. For several providers, this technology will need to be configured and maintained as 3rd party technology which will create operational complications.

When to revisit

After R3 once we have successfully implemented some ingestion workflows to assess value, flexibility, and costs

Edited Sep 29, 2020 by Stephen Whitley (Invited Expert)
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking