ADR: Workflow Service - R3 Improvements
During work with different stream, we identify several critical design issues with Workflow service that needs to be addressed in R3:
- Workflow service is not just an
abstractionover orchestration engine (Airflow) but also contains OSDU specific logic (
UserType). This logic should be moved to Ingestion Service.
- Workflow Service do not respect Data Partitions. Users potentially can trigger any Workflow in the system.
- There is not functionality to register a new Workflow
- Add functionality to register new Workflows
- Add support of Data Partitions
- Remove OSDU specific workflow functionality (
UserType) from Workflow Service.
- Allow OSDU clients directly trigger registered Workflows, without Ingestion Service.
- Update API to reflect Google REST API Design Guide. Please seeOpenAPI Spec for details.
- Accept API changes as a part of R3
- Accept Workflow > Core changes as a part of R3
- Deprecate exiting Workflow API (startWorkflow, etc)
- Registration of workflows required for E2E R3 Ingestion
- API spec is on critical path for CSV Ingestion
- Most of the Core logic changes will be implemented by GCP
- Will require support of CSPs as SPI layer will be touched.
When to revisit
- Post R3
IWorkflowEngineService- Has default implementation. Abstraction over orchestration engine. By default we have implementation for Airflow.
IWorkflowManagerService- Has default implementation. Implements CRUD over Workflow entity.
IWorkflowRunService- - Has default implementation. Implements CRUD over Workflow Run entity.
IWorkflowMetadataRepository- Should be implemented by CSP!. Repository for Workflow entity.
IWorkflowRunRepository- Should be implemented by CSP!. Repository for Workflow Run entity