Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2023-08-18T11:13:31Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/101Update bootstrap names [GONRG-4580]2023-08-18T11:13:31ZMikhail Piatliou (EPAM)Update bootstrap names [GONRG-4580]M11 - Release 0.14Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/100Revert "Merge branch 'cherry-pick-66f449e1' into 'release/0.13'"2022-02-10T08:52:47ZDavid Diederichd.diederich@opengroup.orgRevert "Merge branch 'cherry-pick-66f449e1' into 'release/0.13'"## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/99Revert "Merge branch 'fix-requirements-file' into 'master'"2023-08-18T11:13:32ZDavid Diederichd.diederich@opengroup.orgRevert "Merge branch 'fix-requirements-file' into 'master'"## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M11 - Release 0.14David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/98Cherry-pick 'Adding extra index urls to enable finding these packages' into '...2022-02-09T18:02:47ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Adding extra index urls to enable finding these packages' into 'release/0.13'Original MR: !96Original MR: !96M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/97GONRG-4139 Split dags bootstrap into composer provisioning and DAGs installation2023-08-18T11:13:34ZMaksimelyan Tamashevich (EPAM)GONRG-4139 Split dags bootstrap into composer provisioning and DAGs installation## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Split dags bootstrap job into composer provisioning and DAGs installation jobsM11 - Release 0.14Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/96Adding extra index urls to enable finding these packages2022-02-09T17:16:12ZDavid Diederichd.diederich@opengroup.orgAdding extra index urls to enable finding these packages## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IB...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Adding in the --extra-index-url options to the `requirements.txt` file so this can build out of the boxM10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/95Azure - Add zip file2023-08-18T11:13:35Zharshit aggarwalAzure - Add zip fileM10 - Release 0.13harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/94GONRG-4004: Replace bootstrap values with AF vars2022-02-28T15:01:08ZYan Sushchynski (EPAM)GONRG-4004: Replace bootstrap values with AF vars## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Change bootstrap values to AF variablesM11 - Release 0.14Aleksandr Spivakov (EPAM)Aleksandr Spivakov (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/93Delete osdu-ingest-r2.py2023-08-18T11:13:37ZYan Sushchynski (EPAM)Delete osdu-ingest-r2.py## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Delete deprecated DAGM10 - Release 0.13Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/92[GONRG-4086] Use PyPi package version from master2023-08-18T11:13:39ZDanylo Vanin (EPAM)[GONRG-4086] Use PyPi package version from master## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] I...## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description
Changed bootstrap script for GCP to generally use master version of PyPi packages (ex. 0.13.0.dev1).
For the release of current service - versions of PyPi packages must be manually set to use release version -> ex. 0.12.0
PyPi packages versions are set in `./devops/gcp/composer_requirements.txt`M10 - Release 0.13Danylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/91GONRG-4002: Increase DAG timeout to process large Manifests2023-08-18T11:13:41ZYan Sushchynski (EPAM)GONRG-4002: Increase DAG timeout to process large Manifests## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Sometime the DAG fails on processing 50k records Manifests. It turned out that the DAG has a timeout set to 1 hour. If we increase this value, we are able to process such large ManifestsM10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/90GONRG-3893-bootstrap: added new variables: sa_file_path, data_partition,...2023-08-18T11:13:42ZAliaksandr Lubouski (EPAM)GONRG-3893-bootstrap: added new variables: sa_file_path, data_partition,...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [X] Bootstrap
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [X] Bootstrap
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/88GONRG-3779: Common pipeline for dag2021-12-22T18:57:02ZYan Sushchynski (EPAM)GONRG-3779: Common pipeline for dag## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Create common pipelines for GCP.M10 - Release 0.13Vladislav Shishko (EPAM)Vladislav Shishko (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/87GONRG-3675-bootstrap: added pipeline, Dockerfile, as well as configuration...2023-08-18T11:13:47ZAliaksandr Lubouski (EPAM)GONRG-3675-bootstrap: added pipeline, Dockerfile, as well as configuration...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [x] Deployment Framework Integration
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [x] Deployment Framework Integration
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
GONRG-3675-bootstrap: added pipeline, Dockerfile, as well as configuration files with bootstrap bash scriptM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/86Remove unused dags2021-11-15T05:31:18Zharshit aggarwalRemove unused dagsharshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/85GONRG-3733 remove airflow v1 deployment steps2021-11-11T09:57:52ZAleksandr Spivakov (EPAM)GONRG-3733 remove airflow v1 deployment steps## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M10 - Release 0.13Siarhei Khaletski (EPAM)Yan Sushchynski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/84GONRG-3709 fix E2E tests2021-11-03T08:46:09ZAleksandr Spivakov (EPAM)GONRG-3709 fix E2E tests## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
fix E2E tests for GCP:
- add support for tags in CI steps
- update required services URLs for Community test environmentM10 - Release 0.13Siarhei Khaletski (EPAM)Yan Sushchynski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/83Merge branch 'GONRG-3605_Update_dags_deps_for_release' into 'master'2021-10-27T13:52:46ZSiarhei Khaletski (EPAM)Merge branch 'GONRG-3605_Update_dags_deps_for_release' into 'master'## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M9 - Release 0.12Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/82GONRG-3605: Update deps version to 0.12.02021-10-27T12:45:52ZYan Sushchynski (EPAM)GONRG-3605: Update deps version to 0.12.0## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Update requirements.txt file for Airflow environment to release onesM9 - Release 0.12Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/81Regular expressions for the release branch are incorrect2021-10-26T04:47:36ZDavid Diederichd.diederich@opengroup.orgRegular expressions for the release branch are incorrect## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IB...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
This fixes the `.gitlab-ci.yml` branch expressions. Without this, they are ignored, leading to invalid YAML errors any time a branch named `release/*` is created.M9 - Release 0.12David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.org