Ingestion Workflow merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests2023-08-18T11:17:55Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/207Vuln fix lucene2023-08-18T11:17:55ZGokul NagareVuln fix luceneM10 - Release 0.13Anuj GuptaShrikant GargAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/214Upgrading library versions to incorporate the latest Log4j fix2023-08-18T11:17:49ZDavid Diederichd.diederich@opengroup.orgUpgrading library versions to incorporate the latest Log4j fixThese updates apply version 2.17, addressing CVE-2021-45105
Part of the #135 seriesThese updates apply version 2.17, addressing CVE-2021-45105
Part of the #135 seriesM10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/232Upgrade Log4J to 2.17.12022-08-23T21:25:45ZDavid Diederichd.diederich@opengroup.orgUpgrade Log4J to 2.17.1Closes #138Closes #138M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/225Update GKE SA to follow common naming convention [GONRG-4143]2023-08-18T11:17:39ZMikhail Piatliou (EPAM)Update GKE SA to follow common naming convention [GONRG-4143]M10 - Release 0.13Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/223Update GKE SA to follow common naming convention [GONRG-4143]2023-08-18T11:17:42ZMikhail Piatliou (EPAM)Update GKE SA to follow common naming convention [GONRG-4143]M10 - Release 0.13Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/193Update .gitlab-ci.yml2023-08-18T11:18:06ZAnuj GuptaUpdate .gitlab-ci.ymlM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/200Stabilize Workflow with postman collection (new Approach) (GONRG-3729)2023-08-18T11:17:59ZArtem Dobrynin (EPAM)Stabilize Workflow with postman collection (new Approach) (GONRG-3729)# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How t...# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How to test:
Functionality was tested on GCP with datastore and postgres
# Changes include:
* [ ] Refactor (a non-breaking change that improves code maintainability).
* [X] Bugfix (a non-breaking change that solves an issue).
* [ ] New feature (a non-breaking change that adds functionality).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [x] GCP
* [ ] Azure
* [ ] AWS
* [ ] IBM
* [x] Common code
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [x] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/198Stabilize Workflow with postman collection (new Approach) (GONRG-3729)2021-11-19T12:27:13ZArtem Dobrynin (EPAM)Stabilize Workflow with postman collection (new Approach) (GONRG-3729)# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How t...# Description:
There was an issue while creating system workflow entities. The flag has been ignored, therefore, some logic for system workflow has been skipping.
Also, there was some miswritten exception for workflow not found
# How to test:
Functionality was tested on GCP with datastore and postgres
# Changes include:
* [ ] Refactor (a non-breaking change that improves code maintainability).
* [X] Bugfix (a non-breaking change that solves an issue).
* [ ] New feature (a non-breaking change that adds functionality).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [x] GCP
* [ ] Azure
* [ ] AWS
* [ ] IBM
* [x] Common code
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [x] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Should be merged after https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/191M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/224Revert "Merge branch 'gcp-update-gke-sa' into 'master'"2023-08-18T11:17:41ZOleksandr Kosse (EPAM)Revert "Merge branch 'gcp-update-gke-sa' into 'master'"This reverts merge request !223This reverts merge request !223M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/208log4j vuln fix2023-08-18T11:17:54ZBhushan Radelog4j vuln fix|module pom changes| Ref Issue|
|------|-----------------------------------------------------------------------------------------------|
| IBM | https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/issues...|module pom changes| Ref Issue|
|------|-----------------------------------------------------------------------------------------------|
| IBM | https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/issues/2 |
| Core | https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/54 |
Part of https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/134 seriesM10 - Release 0.13David Diederichd.diederich@opengroup.orgAnuj GuptaBhushan RadeShrikant GargDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/220Ibm core lib upgrade2023-08-18T11:17:44ZAshwani PandeyIbm core lib upgradeIBM core lib version upgrade to rc6IBM core lib version upgrade to rc6M10 - Release 0.13Ashwani PandeyAshwani Pandeyhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/227GONRG-4206: Verify dev2 pipeline for workflow2023-08-18T11:17:36ZNastassia Rabeichykava (EPAM)GONRG-4206: Verify dev2 pipeline for workflowM10 - Release 0.13Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/226GONRG-4118 Add imagePullPolicy value2023-08-18T11:17:37ZIgor Zimovets (EPAM)GONRG-4118 Add imagePullPolicy valueM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/182Gcp workflow osm(GONRG-3404)2021-11-03T09:23:18ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comGcp workflow osm(GONRG-3404)# Description:
Implementation of new common code design based on OSM framework.
# How to test:
Functionality was tested with Datastore and Postgres integration tests passed.
# Changes include:
- [x] Refactor (a non-breaking change that...# Description:
Implementation of new common code design based on OSM framework.
# How to test:
Functionality was tested with Datastore and Postgres integration tests passed.
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [x] New feature (a non-breaking change that adds functionality).
# Changes in:
- [x] GCP
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Migration cannot be done in one step, wanting to keep the current data structure (not model native),
this MR includes several classes that can be removed later:
~~~
`@ConditionalOnProperty(name = "datastore.legacy.data.structure", havingValue = "true")`
~~~
OSM framework is currently **not published** to the community, and the package registry does not have an artifact of it, the build **will fail** until OSM is published.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/221GCP Fix for request with 100k manifest entries (GONRG-4050)2022-01-10T14:12:51ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comGCP Fix for request with 100k manifest entries (GONRG-4050)# Description:
Change airflow request building to reduce heap occupation.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] Ne...# Description:
Change airflow request building to reduce heap occupation.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/212Fix log4j (GONRG-4016)2021-12-21T03:36:41ZArtem Dobrynin (EPAM)Fix log4j (GONRG-4016)Updated version to 2.17.0 to fix vulnerabilities
https://community.opengroup.org/groups/osdu/platform/-/epics/11
Part of the #135 seriesUpdated version to 2.17.0 to fix vulnerabilities
https://community.opengroup.org/groups/osdu/platform/-/epics/11
Part of the #135 seriesM10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/203Enable airflow stable api for gcp2021-11-29T15:31:55ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comEnable airflow stable api for gcp# Description:
Enable stable API for gcp. Minor refactoring & unhardcode integration tests.
~~~
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solv...# Description:
Enable stable API for gcp. Minor refactoring & unhardcode integration tests.
~~~
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M10 - Release 0.13Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/209Disables unsafe subst for log4j2023-08-18T11:17:52ZOleksii Tsyganov (EPAM)Disables unsafe subst for log4jM10 - Release 0.13Oleksii Tsyganov (EPAM)Oleksii Tsyganov (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/201Disable spring web security 2052023-08-18T11:17:57ZBhushan RadeDisable spring web security 205M10 - Release 0.13Anuj GuptaBhushan RadeAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/165Disable accepting dag content based on feature flag [azure]2023-08-18T11:18:27ZAalekh JainDisable accepting dag content based on feature flag [azure]M10 - Release 0.13