Apache AirFlow for ingestion
Using Apache Airflow to support Workflow Orchestration for Ingestion Workflows
Status
- Initiated
- Proposed
- Trialing
- Under review
- Approved
- Retired
Decision
We will use Apache Airflow for implementing and executing ingestion workflows. The will leverage the technology as an orchestration system and AirFlow Operators can be both built in, or developed as custom operators.
Rationale
We need a cross platform workflow orchestration so that we can reuse both ingestion workflows and operators (tasks) in the workflows. This will be an area of high reuse across OSDU members.
Consequences
This is an Open Source solution that will have to be managed within the OSDU Platform. For several providers, this technology will need to be configured and maintained as 3rd party technology which will create operational complications.
When to revisit
After R3 once we have successfully implemented some ingestion workflows to assess value, flexibility, and costs