ADR: Workflow Service - R3 Improvements

Context

During work with different stream, we identify several critical design issues with Workflow service that needs to be addressed in R3:

• Workflow service is not just an abstraction over orchestration engine (Airflow) but also contains OSDU specific logic (DataType, WorkflowType, UserType). This logic should be moved to Ingestion Service.
• Workflow Service do not respect Data Partitions. Users potentially can trigger any Workflow in the system.
• There is not functionality to register a new Workflow

Scope

• Add functionality to register new Workflows
• Add support of Data Partitions
• Remove OSDU specific workflow functionality (DataType, WorkflowType, UserType) from Workflow Service.
• Allow OSDU clients directly trigger registered Workflows, without Ingestion Service.
• Update API to reflect Google REST API Design Guide. Please seeOpenAPI Spec for details.

Decision

• Accept API changes as a part of R3
• Accept Workflow > Core changes as a part of R3
• Deprecate exiting Workflow API (startWorkflow, etc)

Rationale

• Registration of workflows required for E2E R3 Ingestion
• API spec is on critical path for CSV Ingestion

Consequences

• Most of the Core logic changes will be implemented by GCP
• Will require support of CSPs as SPI layer will be touched.

When to revisit

• Post R3

Technical details:

SPI Layer:

• IWorkflowEngineService - Has default implementation. Abstraction over orchestration engine. By default we have implementation for Airflow.
• IWorkflowManagerService - Has default implementation. Implements CRUD over Workflow entity.
• IWorkflowRunService - - Has default implementation. Implements CRUD over Workflow Run entity.
• IWorkflowMetadataRepository - Should be implemented by CSP!. Repository for Workflow entity.
• IWorkflowRunRepository - Should be implemented by CSP!. Repository for Workflow Run entity