csv-parser merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests2024-01-19T11:41:01Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/440standardize azure related dag implementation2024-01-19T11:41:01Zsaketh somarajustandardize azure related dag implementation## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Current Configuration.
- No Azure publish job
- GIT SHA is appended to csv parser dag name, which in turn creates a new dag for each commit
- Dag task image naming convention is not uniform with other repositories.
- Docker image is hardcoded in dag zip artifact.
## Updates description?
- Add Azure publish job to publish dag task image to msosdu ACR.
- GIT SHA appending is removed from to csv parser dag name.
- Update dag task image naming convention to make sure it is uniform with other dag repositories.
- Variabilize docker image in azure dag zip artifact, such that image tag is taken from airflow variables.
Related to https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/merge_requests/1044M23 - Release 0.26saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/439[MSCOSDU-1894] fix logback, reactor-netty-http and json-smart vulnerabilities2024-01-02T08:16:40ZVidyaDharani Lokam[MSCOSDU-1894] fix logback, reactor-netty-http and json-smart vulnerabilities* excluded `logback` dependencies and upgraded `log4j` version for Azure
* upgraded `json-smart` version to `2.5.0` to remediate vulnerability
* upgraded `reactor-netty-http` to `1.1.14` to remediate vulnerability* excluded `logback` dependencies and upgraded `log4j` version for Azure
* upgraded `json-smart` version to `2.5.0` to remediate vulnerability
* upgraded `reactor-netty-http` to `1.1.14` to remediate vulnerabilityM23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/438[ MSCOSDU-1955] remove git sha from dag name2024-01-22T04:33:46Zsaketh somaraju[ MSCOSDU-1955] remove git sha from dag name**Current behavior**
- CSV parser dag name is appended with Git sha of commit which lead to lot of duplicate dags in GLAB env.
**Expected behavior**
- This MR removes appended git sha from dag name, dag file name and in the docker image...**Current behavior**
- CSV parser dag name is appended with Git sha of commit which lead to lot of duplicate dags in GLAB env.
**Expected behavior**
- This MR removes appended git sha from dag name, dag file name and in the docker image.
- This would help in standardizing dags workflows
Changes are tested with https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/jobs/2485855#L293M23 - Release 0.26saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/437[MSCOSDU-1944] upgrade core-lib-azure and clean up pom2024-01-02T11:34:29ZDeepa Kumari[MSCOSDU-1944] upgrade core-lib-azure and clean up pomDue to a change made in core-lib-azure in release 0.19, we need to disable the mdc context logging in order to use the 0.25 release of core-lib-azure.
The change is here:
https://community.opengroup.org/osdu/platform/system/lib/cloud/azu...Due to a change made in core-lib-azure in release 0.19, we need to disable the mdc context logging in order to use the 0.25 release of core-lib-azure.
The change is here:
https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/281#note_274329
So, this is the workaround in CSV parser, that lets us consume the 0.25 version with disabling the Slf4jMDCFilter class.M23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/436[MSCOSDU-1894] upgrade: json, netty versions2023-12-20T15:10:47ZDeepa Kumari[MSCOSDU-1894] upgrade: json, netty versionsM23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/435Cherry-pick 'Full Upgrade of First Party Library Dependencies for Release 0.2...2023-12-16T09:50:34ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Full Upgrade of First Party Library Dependencies for Release 0.25' into release/0.25**Original MR**: !431
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !431
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/pipelines/new?ref=cherry-pick-for-431)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/434Merge branch 'gc-hotfix-image-path' into 'master'2023-12-15T10:53:10ZMikhail Piatliou (EPAM)Merge branch 'gc-hotfix-image-path' into 'master'GONRG-9006: refactored conditions for images paths
See merge request osdu/platform/data-flow/ingestion/csv-parser/csv-parser!433
(cherry picked from commit 14545355923ae83e76aaa4356c8d218db15ed552)
d3f0a5e1 GONRG-9006: refactored cond...GONRG-9006: refactored conditions for images paths
See merge request osdu/platform/data-flow/ingestion/csv-parser/csv-parser!433
(cherry picked from commit 14545355923ae83e76aaa4356c8d218db15ed552)
d3f0a5e1 GONRG-9006: refactored conditions for images pathsM22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/433GONRG-9006: refactored conditions for images paths2023-12-15T10:39:12ZMikhail Piatliou (EPAM)GONRG-9006: refactored conditions for images pathsM22 - Release 0.25Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/432AWS: Cherry pick to release 0.25 - Set the correct user in tests for for user...2023-12-15T10:25:02ZGuillaume CailletAWS: Cherry pick to release 0.25 - Set the correct user in tests for for user context ingestionOriginal MR: https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/430Original MR: https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/430M22 - Release 0.25Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/431Full Upgrade of First Party Library Dependencies for Release 0.252023-12-15T20:09:17ZDavid Diederichd.diederich@opengroup.orgFull Upgrade of First Party Library Dependencies for Release 0.25This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will ...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to try to fully upgrade all dependent libraries to see if the latest code will work.
It is expected that these will often fail, since the upgrades were previously rejected for failing pipelines and have not been directly addressed yet.
This upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
This MR may co-exist with a separate, smaller upgrade MR.
If both pass, this one should be used instead.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: f1fdd409814f5752195a5a00ecee0b553fc017a4
Maven: 0.26.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------ | ------------------ | -------------------------- |
| core-lib-azure | 0.15.0-rc4 | 0.15.0-rc4 |
| core-lib-gc | 0.24.0 | 0.24.0 |
| os-core-lib-aws | 0.21.0 | 0.21.0 |
| obm | 0.21.0 | 0.21.0 |
| oqm | 0.21.0 | 0.21.0 |
| os-core-common | 0.16.0-rc2, 0.24.0 | 0.13.0, 0.16.0-rc2, 0.24.0 |
| os-core-lib-ibm | 0.15.2 | 0.13.0 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: ca68455bc21535c056607fadba2014f7b0cdc14d
Maven: 0.26.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------ | ------------------ | -------------------------- |
| core-lib-azure | 0.15.0-rc4 | 0.15.0-rc4 |
| core-lib-gc | 0.25.0 | 0.25.0 |
| os-core-lib-aws | 0.21.0 | 0.21.0 |
| obm | 0.21.0 | 0.21.0 |
| oqm | 0.21.0 | 0.21.0 |
| os-core-common | 0.16.0-rc2, 0.25.0 | 0.13.0, 0.16.0-rc2, 0.25.0 |
| os-core-lib-ibm | 0.15.2 | 0.13.0 |M22 - Release 0.25https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/430AWS - Set the correct user in tests for for user context ingestion2023-12-15T10:24:25ZGuillaume CailletAWS - Set the correct user in tests for for user context ingestionAWS change only, fix java testAWS change only, fix java testM22 - Release 0.25Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/429Update FOSSA NOTICE2023-12-09T13:06:54ZDavid Diederichd.diederich@opengroup.orgUpdate FOSSA NOTICEThis MR updates the attribution file for the project (also known as the `NOTICE` file).
It is important to keep this up to date to satisfy legal requirements of dependency licenses.
We use FOSSA as the tool to scan for and detect these ...This MR updates the attribution file for the project (also known as the `NOTICE` file).
It is important to keep this up to date to satisfy legal requirements of dependency licenses.
We use FOSSA as the tool to scan for and detect these changes.M22 - Release 0.25https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/428Update gc/render_dag_file.py2023-12-08T13:39:40ZYan Sushchynski (EPAM)Update gc/render_dag_file.pyM22 - Release 0.25https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/427GONRG-8942: fix image paths2023-12-08T08:28:44ZMikhail Piatliou (EPAM)GONRG-8942: fix image pathsM23 - Release 0.26Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/426Fixing s360 vulnerability in CSV Parser DAG2023-12-07T10:20:05ZChahat KumarFixing s360 vulnerability in CSV Parser DAGChahat KumarChahat Kumarhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/425Improve global status messaging resilience2023-12-15T13:37:06ZRiabokon Stanislav(EPAM)[GCP]Improve global status messaging resilienceAdded retry for global status messagingAdded retry for global status messagingM23 - Release 0.26Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/424AWS - Merge from dev2023-11-27T15:17:36ZGuillaume CailletAWS - Merge from devAWS only
* Add or update License header
* Upgrade buildspecAWS only
* Add or update License header
* Upgrade buildspecM22 - Release 0.25Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/423gc: fix k8spod operator2023-12-07T15:12:26ZYan Sushchynski (EPAM)gc: fix k8spod operatorM22 - Release 0.25https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/422Cherry-pick 'Upgrade First Party Library Dependencies for Release 0.24' into ...2023-10-19T06:31:31ZSrinivasan NarayananCherry-pick 'Upgrade First Party Library Dependencies for Release 0.24' into release/0.24**Original MR**: !420
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !420
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/pipelines/new?ref=cherry-pick-for-420)M21 - Release 0.24David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/421Cherry-pick 'Fix jackson-databind-vulnerability' into release/0.242023-10-18T09:20:33ZSrinivasan NarayananCherry-pick 'Fix jackson-databind-vulnerability' into release/0.24**Original MR**: !416
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !416
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/pipelines/new?ref=cherry-pick-for-416)M21 - Release 0.24David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.org