Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2024-01-19T11:40:23Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/164standardize azure related dag implementation2024-01-19T11:40:23Zsaketh somarajustandardize azure related dag implementation- Add publish stage to run azure publish job in release and tag pipelines.
related to https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/merge_requests/1044- Add publish stage to run azure publish job in release and tag pipelines.
related to https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/merge_requests/1044M23 - Release 0.26saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/163Use OIDC Tokens for GC jobs2023-10-17T16:13:45ZAliaksandr Ramanovich (EPAM)Use OIDC Tokens for GC jobsUse GitLab OIDC tokens instead of service account keysUse GitLab OIDC tokens instead of service account keysM22 - Release 0.25Aliaksandr Ramanovich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/162Add workflow rule for CI2023-09-25T08:25:53Zsaketh somarajuAdd workflow rule for CI## Type of change
- [ ] Bug Fix
- [ ] Feature
- [*] CI change
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [*] Azure
...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [*] CI change
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [*] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
- Added workflow rules to configure release and tag pipelines to run on dev secondary environment.
- Related to https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/merge_requests/995M21 - Release 0.24saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/161Cherry-pick 'Update AWS manifest ingestion postman collection path and skip c...2023-09-01T15:38:47ZSrinivasan NarayananCherry-pick 'Update AWS manifest ingestion postman collection path and skip crs conversion and unit conversion' into release/0.23**Original MR**: !159
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !159
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/pipelines/new?ref=cherry-pick-for-159)M20 - Release 0.23David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/160Update devops/gc/override-stages.yml2023-09-01T11:34:27ZYan Sushchynski (EPAM)Update devops/gc/override-stages.yml## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Update the path to the Postman collectionM21 - Release 0.24https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/159Update AWS manifest ingestion postman collection path and skip crs conversion...2023-09-01T11:24:56ZMadalyn MarabellaUpdate AWS manifest ingestion postman collection path and skip crs conversion and unit conversionNOTE: pipeline passed on different branch with identical diff. This branch has the same two commits squashed:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/commit/3309e0c4cb316d5b7cd2494c3d34443438bd1...NOTE: pipeline passed on different branch with identical diff. This branch has the same two commits squashed:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/commit/3309e0c4cb316d5b7cd2494c3d34443438bd1852
## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M20 - Release 0.23Madalyn MarabellaMadalyn Marabellahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/158GONRG-7624: add jobs2023-08-25T08:22:47ZYauheni Rykhter (EPAM)GONRG-7624: add jobsM20 - Release 0.23Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/3Feature/change structure and linters (GONRG-568)2023-08-18T11:15:17ZSiarhei Khaletski (EPAM)Feature/change structure and linters (GONRG-568)The MR brings the new project structure (potentially will be fine-tuned withing new commits).
Added `linters` step for deploy (`pylint` was used)
Some styles for IDE put into `.editorconfig` file (please check if your IDE/Editor supports)The MR brings the new project structure (potentially will be fine-tuned withing new commits).
Added `linters` step for deploy (`pylint` was used)
Some styles for IDE put into `.editorconfig` file (please check if your IDE/Editor supports)M1 - Release 0.1Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/4GONRG-474 Unified CICD and folders structure2023-08-18T11:15:14ZOleksandr Kosse (EPAM)GONRG-474 Unified CICD and folders structureUpdate CICD pipeline for unification.
Update folders structure for unification.Update CICD pipeline for unification.
Update folders structure for unification.M1 - Release 0.1Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/5README.md updates (GONRG-756)2023-08-18T11:15:13ZSiarhei Khaletski (EPAM)README.md updates (GONRG-756)M1 - Release 0.1Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/6R3 Data Ingestion2023-08-18T11:15:11ZSiarhei Khaletski (EPAM)R3 Data IngestionThe MR comes with:
- Implemented R3 Data load;
- Data validation against schemas from Schema Service;
- Common code moved to libs folderThe MR comes with:
- Implemented R3 Data load;
- Data validation against schemas from Schema Service;
- Common code moved to libs folderM1 - Release 0.1Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/7R3 ingestion updates2023-08-18T11:15:09ZSiarhei Khaletski (EPAM)R3 ingestion updatesThe MR comes with:
- GONRG-1457: Stopped replacing fileSource (fix for FileSource)
- GONRG-782: add support custom acl and legal tag. update tests.
- GONRG-1402: Added Logger Section for README.md fileThe MR comes with:
- GONRG-1457: Stopped replacing fileSource (fix for FileSource)
- GONRG-782: add support custom acl and legal tag. update tests.
- GONRG-1402: Added Logger Section for README.md fileM1 - Release 0.1Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/9R3 updates2023-08-18T11:15:08ZSiarhei Khaletski (EPAM)R3 updates## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## What is the current behavior?
- Support of the last R3 Manifest schema
- Refreshed documentation and added sphinx docstring style.
- Added new file service ingestion flow.
## What is the new/expected behavior?
- Publish updates that makes the code vendor agnostic (it will require an additional implementation for each vendor)
## Have you added/updated Unit Tests and Integration Tests?
- [YES]M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/10Feature/providers logic split2023-08-18T11:15:06ZSiarhei Khaletski (EPAM)Feature/providers logic split## Type of change
- [X] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X]...## Type of change
- [X] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X] IBM
## What is the current behavior?
- Decoupled providers logic (please see `src/dags/providers/README.md` file)
- Fixed E2E tests
## Have you added/updated Unit Tests and Integration Tests?
- [YES]M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/11Fix for pre-defined Ids and autogenerated Ids2023-08-18T11:15:05ZSiarhei Khaletski (EPAM)Fix for pre-defined Ids and autogenerated Ids## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## What is the current behavior?
- Fixed issue with pre-defined ids
- Removed Ids autogeneration
## Have you added/updated Unit Tests and Integration Tests?
- [YES]M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/12Fix validation of entity inside the manifest (#31)2023-08-18T11:15:03ZKishore BattulaFix validation of entity inside the manifest (#31)M3 - Release 0.5https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/13GONRG-1709: update workflow service interaction (#32)2023-08-18T11:15:01ZSiarhei Khaletski (EPAM)GONRG-1709: update workflow service interaction (#32)## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [ ] IBM
## What is the current behavior?
- Support the last contract with Workflow Service
## Have you added/updated Unit Tests and Integration Tests?
- [YES]M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/14GONRG-1799: Added Azure requirements2023-08-18T11:15:00ZSiarhei Khaletski (EPAM)GONRG-1799: Added Azure requirements## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## What is the current behavior?
- Added Azure requirements (closes https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/12 conflicts)M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/16ibm-changes2023-08-18T11:14:58ZShrikant Gargibm-changes@wladmirf @jingdongsun @ethiraj@wladmirf @jingdongsun @ethirajM3 - Release 0.5Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/19Fixed issue in environment variable2023-08-18T11:14:57ZKishore BattulaFixed issue in environment variable## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Fixed issue in reading wrong environment variable.M4 - Release 0.7