Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2024-01-19T11:40:23Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/164standardize azure related dag implementation2024-01-19T11:40:23Zsaketh somarajustandardize azure related dag implementation- Add publish stage to run azure publish job in release and tag pipelines.
related to https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/merge_requests/1044- Add publish stage to run azure publish job in release and tag pipelines.
related to https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/merge_requests/1044M23 - Release 0.26saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/163Use OIDC Tokens for GC jobs2023-10-17T16:13:45ZAliaksandr Ramanovich (EPAM)Use OIDC Tokens for GC jobsUse GitLab OIDC tokens instead of service account keysUse GitLab OIDC tokens instead of service account keysM22 - Release 0.25Aliaksandr Ramanovich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/162Add workflow rule for CI2023-09-25T08:25:53Zsaketh somarajuAdd workflow rule for CI## Type of change
- [ ] Bug Fix
- [ ] Feature
- [*] CI change
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [*] Azure
...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [*] CI change
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [*] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
- Added workflow rules to configure release and tag pipelines to run on dev secondary environment.
- Related to https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/merge_requests/995M21 - Release 0.24saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/160Update devops/gc/override-stages.yml2023-09-01T11:34:27ZYan Sushchynski (EPAM)Update devops/gc/override-stages.yml## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Update the path to the Postman collectionM21 - Release 0.24https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/161Cherry-pick 'Update AWS manifest ingestion postman collection path and skip c...2023-09-01T15:38:47ZSrinivasan NarayananCherry-pick 'Update AWS manifest ingestion postman collection path and skip crs conversion and unit conversion' into release/0.23**Original MR**: !159
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !159
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/pipelines/new?ref=cherry-pick-for-159)M20 - Release 0.23David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/159Update AWS manifest ingestion postman collection path and skip crs conversion...2023-09-01T11:24:56ZMadalyn MarabellaUpdate AWS manifest ingestion postman collection path and skip crs conversion and unit conversionNOTE: pipeline passed on different branch with identical diff. This branch has the same two commits squashed:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/commit/3309e0c4cb316d5b7cd2494c3d34443438bd1...NOTE: pipeline passed on different branch with identical diff. This branch has the same two commits squashed:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/commit/3309e0c4cb316d5b7cd2494c3d34443438bd1852
## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M20 - Release 0.23Madalyn MarabellaMadalyn Marabellahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/158GONRG-7624: add jobs2023-08-25T08:22:47ZYauheni Rykhter (EPAM)GONRG-7624: add jobsM20 - Release 0.23Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/156GONRG-7623: pipeline for osdu-ingest-r3-by-reference dag2023-08-14T10:05:50ZMikhail Piatliou (EPAM)GONRG-7623: pipeline for osdu-ingest-r3-by-reference dagM20 - Release 0.23Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/153Cherry-pick 'User context changes for osdu-ingest' into release/0.222023-07-13T13:23:41ZChad LeongCherry-pick 'User context changes for osdu-ingest' into release/0.22**Original MR**: !152
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !152
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/pipelines/new?ref=cherry-pick-for-152)M19 - Release 0.22David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/152User context changes for osdu-ingest2023-07-13T09:24:52ZDeepa KumariUser context changes for osdu-ingestOnly update required: python-libs/ingestion-libs/airflow-libs
No code changes
Added comment for showing file path location from dag code
ADR:https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52Only update required: python-libs/ingestion-libs/airflow-libs
No code changes
Added comment for showing file path location from dag code
ADR:https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/52M19 - Release 0.22Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/151GONRG-6990_fix_versions_of_dags2023-08-18T11:13:09ZYan Sushchynski (EPAM)GONRG-6990_fix_versions_of_dags## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M19 - Release 0.22https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/149Cherry-pick 'Update cicd postman and env' into release/0.212023-05-22T21:51:22ZChad LeongCherry-pick 'Update cicd postman and env' into release/0.21**Original MR**: !148
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !148
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/pipelines/new?ref=cherry-pick-for-148)M18 - Release 0.21David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/148Update cicd postman and env2023-05-22T21:22:12ZAbhishek PatilUpdate cicd postman and env## Type of change
- [ ] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Update cicd postman and envM18 - Release 0.21Okoun-Ola Fabien HouetoAbhishek PatilOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/147Gonrg 7029 anthos to baremetal2023-05-19T09:15:45ZYauheni Rykhter (EPAM)Gonrg 7029 anthos to baremetal## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M18 - Release 0.21Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/146GONRG-7029: update anthos to baremetal2023-05-19T09:03:10ZYauheni Rykhter (EPAM)GONRG-7029: update anthos to baremetalM18 - Release 0.21Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/143Update .gitlab-ci.yml2023-02-02T09:43:51ZYan Sushchynski (EPAM)Update .gitlab-ci.yml## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Skip the unnecessary PyType CICD jobM16 - Release 0.19https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/142GONRG-6369 gc rename2023-02-03T14:17:09ZAliaksandr Ramanovich (EPAM)GONRG-6369 gc renameRename OSDU_GCP to GCRename OSDU_GCP to GCM16 - Release 0.19Aliaksandr Ramanovich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/137[GONRG-6172] Fix postman collection url for GC pipeline2023-08-18T11:13:11ZDanylo Vanin (EPAM)[GONRG-6172] Fix postman collection url for GC pipeline## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates
Fixed postman collection url for GC pipelineM16 - Release 0.19Danylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/138Merge branch 'GONRG-6172-Fix-Postman-Collection-Url' into 'master'2022-12-21T17:30:29ZDanylo Vanin (EPAM)Merge branch 'GONRG-6172-Fix-Postman-Collection-Url' into 'master'## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
Cherry picked from https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/137M15 - Release 0.18Oleksandr Kosse (EPAM)Danylo Vanin (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/134Manifest by reference - DAG add2022-12-02T14:27:27Zshivani karipeManifest by reference - DAG add## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Adding the DAG from manifest-by-reference branch to Master.M15 - Release 0.18Valentin GauthierNaveen RamachandraiahValentin Gauthier