Ingestion Workflow merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests2023-08-18T11:19:43Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/89Fixed issues in delete workflow API2023-08-18T11:19:43ZKishore BattulaFixed issues in delete workflow APIAzure implementation has a flow to deploy the DAG through workflow service. When delete a workflow we should only delete the DAGs that are registered through workflow service. Sending a flag to decide whether to delete a DAG from airflow...Azure implementation has a flow to deploy the DAG through workflow service. When delete a workflow we should only delete the DAGs that are registered through workflow service. Sending a flag to decide whether to delete a DAG from airflow or not.
It has core changes but default to false for isDeployedThroughWorkflowService as other CSPs haven't implemented this feature.M4 - Release 0.7https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/90Get all workflows tenant2023-08-18T11:19:41ZMayank Saggar [Microsoft]Get all workflows tenantAdded a new workflow api with UTs and ITs.
Get All Workflows starting with *prefix* API with rest end point : */workflow/?prefix="test"*
1 Parameters can be specified for this get request namely: *prefix*.
Queries the database for all...Added a new workflow api with UTs and ITs.
Get All Workflows starting with *prefix* API with rest end point : */workflow/?prefix="test"*
1 Parameters can be specified for this get request namely: *prefix*.
Queries the database for all the workflows having the specified prefix. The resulting items are sorted by timestamp in descending order.
ITs for the api:
* Success cases when:
* prefix is provided
* Unauthorized case with no access token and no data access token.
* Unauthorized when given invalid partition idM4 - Release 0.7https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/92Fixed issues in getworkflowRuns API2023-08-18T11:19:37ZKishore BattulaFixed issues in getworkflowRuns APIChecking existence of workflow when getting workflow runsChecking existence of workflow when getting workflow runsM4 - Release 0.7https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/100Workflow Service: Can’t trigger workflow with ‘dagName’ property (GONRG-2056)2023-08-18T11:19:23ZAnastasiia GelmutWorkflow Service: Can’t trigger workflow with ‘dagName’ property (GONRG-2056)# Description:
Fixed triggering workflow with 'dagName' property.
# How to test:
Ingestion Workflow service can be tested with the help of business cases.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code main...# Description:
Fixed triggering workflow with 'dagName' property.
# How to test:
Ingestion Workflow service can be tested with the help of business cases.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] Core
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [ ] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.M5 - Release 0.8Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/101Workflow service failing with Out Of Memory Error (GONRG-2100)2023-08-18T11:19:22ZArtem Dobrynin (EPAM)Workflow service failing with Out Of Memory Error (GONRG-2100)# Description:
Fixed Out of Memory error while loading batch files with big amount of data.
See discussion here: https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/64
# How to test:
Ingestion Workflo...# Description:
Fixed Out of Memory error while loading batch files with big amount of data.
See discussion here: https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/64
# How to test:
Ingestion Workflow service can be tested with the help of business cases.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] Core
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
* [ ] Added Unit Tests, wherever applicable.
* [ ] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.M5 - Release 0.8Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/103[Core] Updated Authorization filter for validating mandatory headers2023-08-18T11:19:18ZAalekh Jain[Core] Updated Authorization filter for validating mandatory headersRefer issue #113Refer issue #113M6 - Release 0.9Aalekh JainAalekh Jainhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/105[Core] POST /workflow - Invalid workflow name - Bad Request2023-08-18T11:19:16ZAalekh Jain[Core] POST /workflow - Invalid workflow name - Bad RequestOriginal issue: #102Original issue: #102M6 - Release 0.9Aalekh JainAalekh Jainhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/106[Core] Added response filter in workflow service2023-08-18T11:19:14ZAalekh Jain[Core] Added response filter in workflow serviceOriginal issue: #99Original issue: #99M6 - Release 0.9https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/107[Core] Updated Delete API to check workflow run status2023-08-18T11:19:13ZMayank Saggar [Microsoft][Core] Updated Delete API to check workflow run statusUpdated Delete api to check workflow run status before sending error when active workflow runs present.Updated Delete api to check workflow run status before sending error when active workflow runs present.M6 - Release 0.9Mayank Saggar [Microsoft]Mayank Saggar [Microsoft]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/111Getting 500 internal server error when try to retrieve to workflow status (GO...2023-08-18T11:19:08ZRiabokon Stanislav(EPAM)[GCP]Getting 500 internal server error when try to retrieve to workflow status (GONRG-2273)# Description:
GET request does not contain a body. Thus, we got 500 internal server error when try to retrieve to workflow status.
Original issue: https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/1...# Description:
GET request does not contain a body. Thus, we got 500 internal server error when try to retrieve to workflow status.
Original issue: https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/164
# How to test:
Trigger the workflow POST https://workflow-drgfbg5txq-uc.a.run.app/v1/workflow/Osdu_ingest/workflowRun
Retrieve the status of the workflow GET [https://workflow-drgfbg5txq-uc.a.run.app/v1/workflow/Osdu_ingest/workflowRun/{{runid}}](https://workflow-drgfbg5txq-uc.a.run.app/v1/workflow/Osdu_ingest/workflowRun/%7B%7Brunid%7D%7D)
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [x] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [x] Azure
- [x] AWS
- [x] IBM
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [ ] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:M5 - Release 0.8Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/136move workflow name validation to core code2023-08-18T11:18:46ZYunhua Koglinmove workflow name validation to core codemove workflow name validation to core codemove workflow name validation to core codeM7 - Release 0.10Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/200Stabilize Workflow with postman collection (new Approach) (GONRG-3729)2023-08-18T11:17:59ZArtem Dobrynin (EPAM)Stabilize Workflow with postman collection (new Approach) (GONRG-3729)# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How t...# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How to test:
Functionality was tested on GCP with datastore and postgres
# Changes include:
* [ ] Refactor (a non-breaking change that improves code maintainability).
* [X] Bugfix (a non-breaking change that solves an issue).
* [ ] New feature (a non-breaking change that adds functionality).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [x] GCP
* [ ] Azure
* [ ] AWS
* [ ] IBM
* [x] Common code
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [x] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/208log4j vuln fix2023-08-18T11:17:54ZBhushan Radelog4j vuln fix|module pom changes| Ref Issue|
|------|-----------------------------------------------------------------------------------------------|
| IBM | https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/issues...|module pom changes| Ref Issue|
|------|-----------------------------------------------------------------------------------------------|
| IBM | https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/issues/2 |
| Core | https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/54 |
Part of https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/134 seriesM10 - Release 0.13David Diederichd.diederich@opengroup.orgAnuj GuptaBhushan RadeShrikant GargDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/247[Core] Improve code coverage2023-08-18T11:17:20ZAalekh Jain[Core] Improve code coverageAttached below is the update state of code coverage in workflow core.
![ss](/uploads/da0b25907af7c1a793c904f5c50ed3b5/ss.png)Attached below is the update state of code coverage in workflow core.
![ss](/uploads/da0b25907af7c1a793c904f5c50ed3b5/ss.png)M12 - Release 0.15harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/251Version endpoint has incorrect path (GONRG-4432)2023-08-18T11:17:17ZAnastasiia GelmutVersion endpoint has incorrect path (GONRG-4432)## Type of change
- [x] Bug Fix
- [ ] Feature
Issue link: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/141
## Does this introduce a change in the core logic?
- [NO]
## Does this introd...## Type of change
- [x] Bug Fix
- [ ] Feature
Issue link: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/141
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Does this introduce a breaking change?
- [YES]M11 - Release 0.14Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/266Resolve "Status publisher incorrectly sets status to FAILED"2023-08-18T11:17:02ZMorris EstepaResolve "Status publisher incorrectly sets status to FAILED"Closes #143Closes #143M12 - Release 0.15Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/461Cherry-pick 'Adding provider specific artifact versions' into release/0.222023-07-28T10:40:03ZChad LeongCherry-pick 'Adding provider specific artifact versions' into release/0.22**Original MR**: !460
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !460
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/pipelines/new?ref=cherry-pick-for-460)M19 - Release 0.22David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/460Adding provider specific artifact versions2023-07-28T10:23:48ZDavid Diederichd.diederich@opengroup.orgAdding provider specific artifact versionsThis enables provider specific releases, incrementing the version of only one componentThis enables provider specific releases, incrementing the version of only one componentM19 - Release 0.22David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/378Update vulnerable lib versions2023-07-25T07:11:22ZManish JangidUpdate vulnerable lib versions* Updating versions of vulnerable libraries
* https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/123
* https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/124
* https...* Updating versions of vulnerable libraries
* https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/123
* https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/124
* https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/125
* https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/130M16 - Release 0.19https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/457Cherry-pick '[Dependency Updates] WhiteSource fixes [jul2023]' into release/0.222023-07-13T13:23:00ZChad LeongCherry-pick '[Dependency Updates] WhiteSource fixes [jul2023]' into release/0.22**Original MR**: !455
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !455
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/pipelines/new?ref=cherry-pick-for-455)M19 - Release 0.22David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.org