Ingestion Workflow merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests2022-02-04T19:53:36Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/236Cherry-pick 'fix json-smart version' into 'release/0.13'2022-02-04T19:53:36ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'fix json-smart version' into 'release/0.13'Original MR: !235Original MR: !235M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/232Upgrade Log4J to 2.17.12022-08-23T21:25:45ZDavid Diederichd.diederich@opengroup.orgUpgrade Log4J to 2.17.1Closes #138Closes #138M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/227GONRG-4206: Verify dev2 pipeline for workflow2023-08-18T11:17:36ZNastassia Rabeichykava (EPAM)GONRG-4206: Verify dev2 pipeline for workflowM10 - Release 0.13Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/213Code coverage2023-08-18T11:17:50ZAalekh JainCode coverageTasks accomplished as part of this MR
1. Added the support for generating code coverage reports using `Jacoco`
2. Updated the settings for CI/CD pipelines (Under section General Pipelines) to capture the code coverage using the Test Cov...Tasks accomplished as part of this MR
1. Added the support for generating code coverage reports using `Jacoco`
2. Updated the settings for CI/CD pipelines (Under section General Pipelines) to capture the code coverage using the Test Coverage Parsing regex as: `Total.*?([0-9]{1,3})%`
3. Added code coverage badge for azure in the `README.md` corresponding to azure cloud provider.
Attached below are the screenshots of the code coverage captured through bade and in the pipeline stages -
1. ![image](/uploads/4a53a8708293ed392a535e2d0186f455/image.png)
2. ![image](/uploads/4e989b4ccad8ab35203cfb634a70b2ed/image.png)M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/226GONRG-4118 Add imagePullPolicy value2023-08-18T11:17:37ZIgor Zimovets (EPAM)GONRG-4118 Add imagePullPolicy valueM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/217Azure deploy fixes2023-08-18T11:17:47ZAalekh JainAzure deploy fixesM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/225Update GKE SA to follow common naming convention [GONRG-4143]2023-08-18T11:17:39ZMikhail Piatliou (EPAM)Update GKE SA to follow common naming convention [GONRG-4143]M10 - Release 0.13Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/224Revert "Merge branch 'gcp-update-gke-sa' into 'master'"2023-08-18T11:17:41ZOleksandr Kosse (EPAM)Revert "Merge branch 'gcp-update-gke-sa' into 'master'"This reverts merge request !223This reverts merge request !223M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/223Update GKE SA to follow common naming convention [GONRG-4143]2023-08-18T11:17:42ZMikhail Piatliou (EPAM)Update GKE SA to follow common naming convention [GONRG-4143]M10 - Release 0.13Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/207Vuln fix lucene2023-08-18T11:17:55ZGokul NagareVuln fix luceneM10 - Release 0.13Anuj GuptaShrikant GargAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/219Airflow2 upgrade doc2023-08-18T11:17:45ZSaurabh TripathiAirflow2 upgrade docM10 - Release 0.13Vineeth Guna [Microsoft]Vineeth Guna [Microsoft]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/221GCP Fix for request with 100k manifest entries (GONRG-4050)2022-01-10T14:12:51ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comGCP Fix for request with 100k manifest entries (GONRG-4050)# Description:
Change airflow request building to reduce heap occupation.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] Ne...# Description:
Change airflow request building to reduce heap occupation.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/220Ibm core lib upgrade2023-08-18T11:17:44ZAshwani PandeyIbm core lib upgradeIBM core lib version upgrade to rc6IBM core lib version upgrade to rc6M10 - Release 0.13Ashwani PandeyAshwani Pandeyhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/199[Azure] Limiting trigger workflow requests in workflow ingestion service2023-08-18T11:18:00ZAalekh Jain[Azure] Limiting trigger workflow requests in workflow ingestion serviceM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/197[Azure] Add support for caching in workflow ingestion service2023-08-18T11:18:02ZAalekh Jain[Azure] Add support for caching in workflow ingestion serviceM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/214Upgrading library versions to incorporate the latest Log4j fix2023-08-18T11:17:49ZDavid Diederichd.diederich@opengroup.orgUpgrading library versions to incorporate the latest Log4j fixThese updates apply version 2.17, addressing CVE-2021-45105
Part of the #135 seriesThese updates apply version 2.17, addressing CVE-2021-45105
Part of the #135 seriesM10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/212Fix log4j (GONRG-4016)2021-12-21T03:36:41ZArtem Dobrynin (EPAM)Fix log4j (GONRG-4016)Updated version to 2.17.0 to fix vulnerabilities
https://community.opengroup.org/groups/osdu/platform/-/epics/11
Part of the #135 seriesUpdated version to 2.17.0 to fix vulnerabilities
https://community.opengroup.org/groups/osdu/platform/-/epics/11
Part of the #135 seriesM10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/209Disables unsafe subst for log4j2023-08-18T11:17:52ZOleksii Tsyganov (EPAM)Disables unsafe subst for log4jM10 - Release 0.13Oleksii Tsyganov (EPAM)Oleksii Tsyganov (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/208log4j vuln fix2023-08-18T11:17:54ZBhushan Radelog4j vuln fix|module pom changes| Ref Issue|
|------|-----------------------------------------------------------------------------------------------|
| IBM | https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/issues...|module pom changes| Ref Issue|
|------|-----------------------------------------------------------------------------------------------|
| IBM | https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/issues/2 |
| Core | https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/54 |
Part of https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/134 seriesM10 - Release 0.13David Diederichd.diederich@opengroup.orgAnuj GuptaBhushan RadeShrikant GargDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/204Add a version of Airflow into an endpoint 'info' for Workflow Service(GONRG-3...2021-12-15T19:43:19ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comAdd a version of Airflow into an endpoint 'info' for Workflow Service(GONRG-3777)# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be reques...# Description:
Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/132 <br/>
Add a version of Airflow into an endpoint 'info' for Workflow Service.<br/>
A version of Airflow will be requested only when enabled stable API `osdu.airflow.version2=true`.<br/>
# How to test:
Functionality can be testes with a request:
~~~
curl --location --request GET 'https://com.osdu.club/api/workflow/info' \
~~~
Response example:
~~~
{
"groupId": "org.opengroup.osdu",
"artifactId": "workflow-gcp",
"version": "0.13.0-SNAPSHOT",
"buildTime": "2021-11-25T12:29:41.581Z",
"branch": "add-airflow-version-for-info-endpoint",
"commitId": "5e0816ea1088e6205fae839db389ebf403dc7885",
"commitMessage": "Add airflow version for info endpoint, enable stable API for gcp",
"connectedOuterServices": [
{
"name": "Airflow",
"version": "2.1.2+composer"
}
]
}
~~~
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [ ] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]