Skip to content
GitLab
Explore
Sign in
Register
This is an archived project. Repository and other project resources are read-only.
OSDU Software
Documentation
Wiki
Osdu (c)
Design and implementation
Ingestion and enrichment detail
R3 Plug And Play Workflow For Ingestion
R3 Plug And Play Workflow For Ingestion
· Changes
Page history
Update R3 Plug And Play Workflow For Ingestion
authored
Jul 08, 2020
by
Dania Kodeih (Microsoft)
Hide whitespace changes
Inline
Side-by-side
OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R3-Plug-And-Play-Workflow-For-Ingestion.md
View page @
72b68b48
...
@@ -152,7 +152,14 @@ This picture shows how each workflow was configured in the 3 implementations:
...
@@ -152,7 +152,14 @@ This picture shows how each workflow was configured in the 3 implementations:
# Recommendation
# Recommendation
Abstract away the workflow components and create a "Plug-and-play" workflow architecture as follows.
Abstract away the workflow components and create a "Plug-and-play" workflow architecture as follows:
### Workflow service should be Workflow engine agnostic (to support Plug-n-Play architecture)
-
Remove airflowRunId property from WorkflowStatus class in org.opengroup.osdu.workflow.model
-
Workflow service and workflow engine two-way communications via workflowId generated by Workflow service core implementation
-
Workflow engine to update workflow status by calling Workflow service with workflowId (e.g. finished, failed, etc.)
-
The committee should contribute and leverage shared Python Libraries for transformation and enrichment business logic
Add Workflow client in
[
osdu_api Python SDK
](
https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk
)
(On July 6th, the PMC voted to go ahead with Airflow as the default implementation for now, so this recommendation will not be adopted)
(On July 6th, the PMC voted to go ahead with Airflow as the default implementation for now, so this recommendation will not be adopted)
...
...