Ingestion Workflow merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests2022-07-26T15:08:22Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/234Getting workflow status periodically returns 500 error because of an unknown ...2022-07-26T15:08:22ZAnastasiia GelmutGetting workflow status periodically returns 500 error because of an unknown state (GONRG-4248)## Type of change
- [x] Bug Fix
- [ ] Feature
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/140
---
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a c...## Type of change
- [x] Bug Fix
- [ ] Feature
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/140
---
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Does this introduce a breaking change?
- [NO]
## What is the new/expected behavior?
Added new DagState.
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_dag_run
## Have you added/updated Unit Tests and Integration Tests?
noM11 - Release 0.14Riabokon Stanislav(EPAM)[GCP]Andrei Dalhikh [EPAM/GC]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/144Add GSM2021-08-30T18:28:57ZMaksim MalkovAdd GSM[issue#126](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/126)
## Contents
* bugfixes
* GSM integration
### GSM implementation details
In the current iteration, we implemented both servic...[issue#126](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/126)
## Contents
* bugfixes
* GSM integration
### GSM implementation details
In the current iteration, we implemented both services for publishing `STATUS` types of messages. For actual use, we have integrated only the status part according to our architectural inputs.
After this MR will be merged we can expect the following messages will be sent(or at least has an attempt of in case of resource absence):
- Once a user called `triggerWorkflow` service will send GSM with `SUBMITTED` status
- Once a user called `updateWorkflowRun` service will send GSM with `<Any of logically correct and supported>` status retrieved from update payload (we'll send a message only if status was changed)
Currently, we are providing all provide with default dull implementation of Message Sender(`IEventPublisher`) it can be easily overridden as we did for Azure provider.
## About GSM
### Introduction <a name="introduction"></a>
Global Status monitoring is a mechanism to track the status of data journey/dataflows on the data platform.
The infrastructure would help in tracking the status of file/data/record ingested through File Service/ Storage API/ Specific DOMS until it is consumed by dependent services.
Every stage publishes one status message to the message queue. From there `Status Collector` picks up messages and normalizes them to store them in persistent storage for future reference. Then `Status Processor` provides an API to query and check the status of past datasets.
### Status Data Model <a name="dataModel"></a>
Data Model properties help any user to search for status with multiple or specific properties. Every request will be tracked through specific `dataSetId` & its associated `correlationId`.
Status Data Model is being distributed across multiple tables for tracking whether dataflow has finished or not and if it is Successful or Failed.
One table holds DataSet Details and another table holds the overall Status of that dataflow journey.
* **DataSet Details** - Dataset can be anything that contains data, for e.g., File is one of the types of datasets which contains data inside, File Collection could be another dataset that would contain a set of files.
* **Status** - hold the overall status of that data flow. `correlationId` is used as a unique id to capture a single request going through different stages of our Data Platform.
### How to publish status and dataset details events <a name="howToPublishStatusEvents"></a>
Any service which wants to publish status and dataset details have to follow the below steps:
1. **Add `core common lib` as dependency** - There are models, classes, and interface defined in `core common lib` from Azure.
We have to make sure we have selected the right version of the library which includes these classes.
2. **All possible scenarios to publish Status/Dataset Details** - It is advised to find out all possible scenarios in which either Status or Dataset Details can be published.
A service can publish multiple sets of both Status and Dataset Details.
3. **Cloud Implementation to publish Status/Dataset Details** - You need to provide an implementation of `IEventPublisher` interface from core common lib.
Publish method in this interface accepts an array of Messages and Maps of string attributes. The message is an interface implemented by both Status and Dataset Details.
So this method expects an array of either Status or Dataset Details. This method of `IEventPublisher` has to be implemented with cloud-specific codes to publish events in `statuschangedtopic`.
Note: We have Azure implementation of Global Status Monitoring. Services that are not part of OSDU AKS cluster have to use /status and /datasetDetails endpoints of `Status Processor` service. `Status Processor` service will publish status and dataset details in `statuschangedtopic`.
#### Sample of status and dataset details message <a name="sample"></a>
The status messages are one of two kinds - DataSet Details and Status, but they are published into the same `statuschangedtopic`.
* **DataSet Details**
```json
[
{
"kind": "datasetDetails",
"properties": {
"correlationId": "12345",
"datasetId": "12345",
"datasetVersionId": "1",
"datasetType": "FILE",
"recordCount": 10,
"timestamp": 1625221800
}
}
]
```
* **Status**
```json
[
{
"kind": "status",
"properties": {
"correlationId": "12345",
"recordId": "12334",
"recordIdVersion": "123ff",
"stage": "STORAGE_SYNC",
"status": "FAILED",
"message": "acl is not valid",
"errorCode": 400,
"userEmail": "test@email.com",
"timestamp": 1625221800
}
}
]
```
#### Core Common Library contents for GSM <a name="coreCommonLibContents"></a>
1. **Models** - `StatusDetails` and `DatasetDetails` - These 2 models should be used to publish status and dataset details.
2. **Utility** - `AttributesBuilder` - This will help to create an attributes map which is required in publishing method of `IEventPublisher` to publish status or dataset details. Attributes map will consist of `data partition id` and `correlation id`.
3. **Publisher Interface** - `IEventPublisher` - This is the interface that a cloud provider has to implement to produce status and dataset details. It contains a method that accepts the Message array and Attributes maps. The message is an interface implemented by both Status and Dataset Details.
### Supported Stages and Statuses <a name="stagesAndStatuses"></a>
#### Stages and Services Mapping
| Stage | Service |
|-------|---------|
| DATASET_SYNC | File Service, Dataset |
| INGESTOR | All Ingestors for e.g., CSV, LAS/DLIS/Document |
| INGESTOR_SYNC | All Ingestors for e.g., CSV, LAS/DLIS/Document |
| WKS_SYNC | All those services that create WKS source records in the Data Platform, for e.g., WKS Transformation Service |
| WKE_SYNC | WKE Service |
| STORAGE_SYNC | Storage Service |
| ES_SYNC | Indexer Service |
#### Supported Statuses
| Status |
|--------|
| SUBMITTED |
| SUCCESS |
| FAILED |
| IN_PROGRESS |
| SKIPPED |
| PARTIAL_SUCCESS |M8 - Release 0.11https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/103[Core] Updated Authorization filter for validating mandatory headers2023-08-18T11:19:18ZAalekh Jain[Core] Updated Authorization filter for validating mandatory headersRefer issue #113Refer issue #113M6 - Release 0.9Aalekh JainAalekh Jainhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/82IBM implementation2023-08-18T11:19:51ZBhushan RadeIBM implementationIBM Changes :
- Added IBM Implementation
Core changes :
- one change in **worflow-core** at /workflow-core/src/main/java/org/opengroup/osdu/workflow/service/**AirflowWorkflowEngineServiceImpl.java**
Issue link for core change - https:...IBM Changes :
- Added IBM Implementation
Core changes :
- one change in **worflow-core** at /workflow-core/src/main/java/org/opengroup/osdu/workflow/service/**AirflowWorkflowEngineServiceImpl.java**
Issue link for core change - https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/87M4 - Release 0.7Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/532Java17migration2024-03-27T09:51:53ZVikas Hoode [BP]vikas.hoode@bp.comJava17migrationjdk 17 upgradejdk 17 upgradeM23 - Release 0.26Vikas Hoode [BP]vikas.hoode@bp.comVikas Hoode [BP]vikas.hoode@bp.comhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/517Solxget/vulnerablity2024-01-24T16:05:18ZSolomon AyalewSolxget/vulnerablityThe following PR addresses the following vulnerabilites.
| 17 | CVE-2022-31159 | HIGH | com.amazonaws:aws-java-sdk-s3-1.11.1018 | Java | os_ingestion_workflow |
|----|---------------------|------|--------------------...The following PR addresses the following vulnerabilites.
| 17 | CVE-2022-31159 | HIGH | com.amazonaws:aws-java-sdk-s3-1.11.1018 | Java | os_ingestion_workflow |
|----|---------------------|------|------------------------------------------------------|------|-----------------------|
| 18 | CVE-2022-42003 | HIGH | com.fasterxml.jackson.core:jackson-databind-2.13.2.2 | Java | os_ingestion_workflow |
| 19 | CVE-2022-42004 | HIGH | com.fasterxml.jackson.core:jackson-databind-2.13.2.2 | Java | os_ingestion_workflow |
| 20 | GHSA-xpw8-rcwv-8f8p | HIGH | io.netty:netty-codec-http2-4.1.70.Final | Java | os_ingestion_workflow |
| 21 | CVE-2023-46589 | HIGH | org.apache.tomcat.embed:tomcat-embed-core-9.0.76 | Java | os_ingestion_workflow |
| 22 | CVE-2023-31582 | HIGH | org.bitbucket.b_c:jose4j-0.7.0 | Java | os_ingestion_workflow |
| 23 | CVE-2021-37714 | HIGH | org.jsoup:jsoup-1.12.1 | Java | os_ingestion_workflow |M23 - Release 0.26https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/515[MSCOSDU-1894] upgrade json-smart and reactor-netty2024-01-05T16:31:24ZVidyaDharani Lokam[MSCOSDU-1894] upgrade json-smart and reactor-netty* upgraded `json-smart` to `2.5.0` to remediate vulnerability.
* upgraded `reactor-netty` to `1.1.14`.* upgraded `json-smart` to `2.5.0` to remediate vulnerability.
* upgraded `reactor-netty` to `1.1.14`.M23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/511Use full URL instead of relative path2024-02-06T09:40:21ZLawrence ChanUse full URL instead of relative path* Issue Reference: https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/issues/68
* Added configuration `api.server.fullUrl.enabled` to enable full server url in OpenAPI swagger
* Currently only in Azure it is en...* Issue Reference: https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/issues/68
* Added configuration `api.server.fullUrl.enabled` to enable full server url in OpenAPI swagger
* Currently only in Azure it is enabled. For Other \[CSP/Common Core\] there is no change.
### Configuration Details
* `api.server.fullUrl.enabled=true` It will generate full server url in the OpenAPI swagger
* `api.server.fullUrl.enabled=false` It will generate only the contextPath
* Reference: https://springdoc.org/faq.html#_how_is_server_url_generatedM23 - Release 0.26https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/510[MSCOSDU-1894] fix spring-security-vulnerabilities2023-12-27T08:39:17ZVidyaDharani Lokam[MSCOSDU-1894] fix spring-security-vulnerabilities* Issue Reference: https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/185
* Fix CVE-2023-34034
* Upgrade `spring-security-core` to `5.8.8`* Issue Reference: https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/185
* Fix CVE-2023-34034
* Upgrade `spring-security-core` to `5.8.8`M23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/508Cherry-pick 'Full Upgrade of First Party Library Dependencies for Release 0.2...2023-12-16T09:50:45ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Full Upgrade of First Party Library Dependencies for Release 0.25' into release/0.25**Original MR**: !504
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !504
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/pipelines/new?ref=cherry-pick-for-504)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/505Implement Airflow facade endpoint2024-01-08T10:10:27ZRiabokon Stanislav(EPAM)[GCP]Implement Airflow facade endpointhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/159
# How to test:
Postman and integration tests.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainabilit...https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/159
# How to test:
Postman and integration tests.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GC
- [x] Azure
- [x] AWS
- [x] IBM
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M23 - Release 0.26Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/504Full Upgrade of First Party Library Dependencies for Release 0.252023-12-15T20:09:01ZDavid Diederichd.diederich@opengroup.orgFull Upgrade of First Party Library Dependencies for Release 0.25This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will ...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will often fail, since the upgrades were previously rejected for failing pipelines and have not been directly addressed yet.
This upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
This MR may co-exist with a separate, smaller upgrade MR.
If both pass, this one should be used instead.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 5d7181d2567c65708bba59cc912dabbabd46f0ae
Maven: 0.26.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | --------------------- |
| core-lib-azure | 0.24.0 | 0.0.28 |
| core-lib-gc | 0.24.0 | |
| os-core-lib-aws | 0.23.0 | 0.23.0 |
| oqm | 0.24.0 | |
| os-core-common | 0.24.0 | 0.24.0, 0.23.0 |
| os-core-lib-ibm | 0.24.0 | 0.24.0 |
| osm | 0.24.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.5, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.10.1 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-slf4j-impl | 2.17.1 | 2.12.1 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.28, 5.3.24, 5.3.22 | 5.2.2.RELEASE, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.25, 1.30 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: 9a416eb3e8df24830cc9232b808d27e6048524ae
Maven: 0.26.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | --------------------- |
| core-lib-azure | 0.25.0 | 0.0.28 |
| core-lib-gc | 0.25.0 | |
| os-core-lib-aws | 0.23.0 | 0.23.0 |
| oqm | 0.25.0 | |
| os-core-common | 0.25.0 | 0.25.0, 0.23.0 |
| os-core-lib-ibm | 0.25.0 | 0.25.0 |
| osm | 0.25.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.5, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.10.1 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-slf4j-impl | 2.17.1 | 2.12.1 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.28, 5.3.24, 5.3.22 | 5.2.2.RELEASE, 5.3.30 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.25, 1.30 |M22 - Release 0.25https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/503[MSCOSDU-1851] Upgrade core-common, azure and netty2023-12-13T15:04:32ZDeepa Kumari[MSCOSDU-1851] Upgrade core-common, azure and netty1. core-lib-azure: from 0.24.0 to 0.25.0-rc2
1. os-core-common: from 0.24.0 to 0.25.0-rc3
1. org.json: from 20220924 to 20231013
1. netty: from 4.1.98 to 4.1.1011. core-lib-azure: from 0.24.0 to 0.25.0-rc2
1. os-core-common: from 0.24.0 to 0.25.0-rc3
1. org.json: from 20220924 to 20231013
1. netty: from 4.1.98 to 4.1.101M23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/497Pass workflow user ID to the Airflow as part of payload (GONRG-8838)2023-11-23T11:23:09ZRiabokon Stanislav(EPAM)[GCP]Pass workflow user ID to the Airflow as part of payload (GONRG-8838)# Description:
Added a user-email in exec_context to trigger workflow under this user.
# How to test:
Impersonation case, int tests.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] ...# Description:
Added a user-email in exec_context to trigger workflow under this user.
# How to test:
Impersonation case, int tests.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] Common code
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M22 - Release 0.25Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/496add user ID to airflow as part of payload in execution_context2023-12-15T10:22:14ZVidyaDharani Lokamadd user ID to airflow as part of payload in execution_contextRelated issues: [!157](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/157) , [!158](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/158)
- A...Related issues: [!157](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/157) , [!158](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/158)
- As discussed in the above issues we need to remove the 'x-user-id' header from core as we might want to use the service without a service mesh or similar infrastructure.
- As the existing azure code relies on 'x-user-id' header this MR includes the change removing the 'x-user-id' from core part and move to azure provider level.
Original issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52M22 - Release 0.25VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/493Cherry-pick 'Full Upgrade of First Party Library Dependencies' into release/0.242023-10-20T12:16:21ZChad LeongCherry-pick 'Full Upgrade of First Party Library Dependencies' into release/0.24**Original MR**: !489
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !489
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/pipelines/new?ref=cherry-pick-for-489)M21 - Release 0.24David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/489Full Upgrade of First Party Library Dependencies2023-10-20T12:15:40ZChad LeongFull Upgrade of First Party Library DependenciesThis generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep all dependent libraries up to date.
This upgrade can be merged immediately without further approval if the C...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep all dependent libraries up to date.
This upgrade can be merged immediately without further approval if the CI pipeline reports success.
If this MR has failed, we need to work with the maintainers and affected provider teams to find a solution.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 8848dd0cbbd228abecc2722c9e90c51bef3157c4
Maven: 0.25.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | --------------------- |
| core-lib-azure | 0.23.2 | 0.0.28 |
| core-lib-gc | 0.23.1 | |
| os-core-lib-aws | 0.23.0 | 0.23.0 |
| oqm | 0.23.0 | |
| os-core-common | 0.23.3 | 0.23.3, 0.23.0 |
| os-core-lib-ibm | 0.24.0 | 0.23.0 |
| osm | 0.23.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.5, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.10.1 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-slf4j-impl | 2.17.1 | 2.12.1 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.28, 5.3.24, 5.3.22 | 5.2.2.RELEASE, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.25, 1.30 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade-2
SHA: 622bf2980eba9c993c098037d03fa2c5ba02e674
Maven: 0.25.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | --------------------- |
| core-lib-azure | 0.24.0 | 0.0.28 |
| core-lib-gc | 0.24.0 | |
| os-core-lib-aws | 0.23.0 | 0.23.0 |
| oqm | 0.24.0 | |
| os-core-common | 0.24.0 | 0.24.0 |
| os-core-lib-ibm | 0.24.0 | 0.24.0 |
| osm | 0.24.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.5, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.10.1 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.12.1 |
| (3rd Party) org.apache.logging.log4j.log4j-slf4j-impl | 2.17.1 | 2.12.1 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.28, 5.3.24, 5.3.22 | 5.2.2.RELEASE, 5.3.22 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.25, 1.30 |M21 - Release 0.24https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/462OpenAPI 3.0 Documentation using springdoc2023-08-18T13:06:20ZJayesh BagulOpenAPI 3.0 Documentation using springdoc**Link to ADR(Architecture Decision Record)** : [Swagger using springdoc-openapi](https://community.opengroup.org/osdu/platform/system/home/-/issues/97)
## OpenAPI 3.0 related changes
* upgraded to latest **springdoc openapi** latest v...**Link to ADR(Architecture Decision Record)** : [Swagger using springdoc-openapi](https://community.opengroup.org/osdu/platform/system/home/-/issues/97)
## OpenAPI 3.0 related changes
* upgraded to latest **springdoc openapi** latest version [1.7.0](https://mvnrepository.com/artifact/org.springdoc/springdoc-openapi-ui/1.6.14)
* Documented the below API's with OpenAPI 3.0 **Annotations**
- WorkflowManager API
- WorkflowRun API
- WorkflowSystemManager API
- HealthCheck API
- Info API
* Added the standard HTTP Response(4xx, 5x\*\*\*\*x) for API Responses
* Custom Path for
* **Swagger UI**: https://host/context-path/swagger (will redirect to https://host/context-path/swagger-ui/index.html)
* **api-docs (JSON)** : https://host/context-path/api-docs
* **api-docs (YAML)** : https://host/context-path/api-docs.yaml
* Azure Swagger GLAB(for Reference)
* **Swagger UI**: https://osdu-glab.msft-osdu-test.org/api/workflow/swagger (will redirect to https://osdu-glab.msft-osdu-test.org/api/workflow/swagger-ui/index.html)
* **api-docs (JSON)** : https://osdu-glab.msft-osdu-test.org/api/workflow/api-docs
* **api-docs (YAML)** :https://osdu-glab.msft-osdu-test.org/api/workflow/api-docs.yaml
* Marked the below **Internal** API's as **Hidden**
- Azure WhoamiController
- Azure CustomOperatorApi
## Other Changes
- **Configurable** descriptions managed in [swagger.properties](https://community.opengroup.org/osdu/platform/system/search/-/blob/az/td-oas/search-core/src/main/resources/swagger.properties)
- Deleted [HomeController, HomeControllerTest, SpringfoxSwaggerHostResolver, SpringfoxSwaggerHostResolverTest,
SwaggerDocumentationConfig]
- Updated Readme for swagger related information
## References
- https://springdoc.org/faq.html#_can_i_use_spring_property_with_swagger_annotations
- https://springdoc.org/migrating-from-springfox.htmlM20 - Release 0.23Jayesh BagulJayesh Bagulhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/461Cherry-pick 'Adding provider specific artifact versions' into release/0.222023-07-28T10:40:03ZChad LeongCherry-pick 'Adding provider specific artifact versions' into release/0.22**Original MR**: !460
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !460
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/pipelines/new?ref=cherry-pick-for-460)M19 - Release 0.22David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/460Adding provider specific artifact versions2023-07-28T10:23:48ZDavid Diederichd.diederich@opengroup.orgAdding provider specific artifact versionsThis enables provider specific releases, incrementing the version of only one componentThis enables provider specific releases, incrementing the version of only one componentM19 - Release 0.22David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.org