Orchestration and Ingestion Services and API specification
OpenDES requires an ingestion service to ingest data and enrich its metadata. The Orchestration and Ingestion flow:
- Provides an ingestion flow which unifies OSDU R1 and DELFI Data Ecosystem requirements
- Refactors the orchestration policy from the DELFI Data Ecosystem application code into a workflow engine
- Under review
Context & Scope
Current DELFI orchestration is explicit: each service publishes an event, which is expected by the next service
- There’s no way to inject custom logic into the process
- There’s no way to customize existing process for particular client/data type
We need a mechanism to support agile ingestion workflow development which supports different data types and enrichment requirements.
The scope of this decision is to agree on the services and their APIs
When to revisit
Tradeoff Analysis - Input to decision
Without these services and APIs, there isn't an API, nor a standardized mechanism to ingest data; instead there is a collection of ad-hoc approaches and scripts.
Some operators which to test OSDU by ingesting data from a number of systems, and then consuming it in others. Half of this use case is unaddressed without an Ingestion flow.
Alternatives and implications
- The Orchestration and Ingestion flow proposes and trials an approach.
- No alternatives have been proposed.
Decision criteria and tradeoffs