Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2021-10-12T15:12:21Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/78GONRG-3452: Move Ingestion Logic from SDK2021-10-12T15:12:21ZYan Sushchynski (EPAM)GONRG-3452: Move Ingestion Logic from SDK## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Updates description?
Move Ingestion Logic from Python SDK into separate package.M9 - Release 0.12Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/77Azure - pipeline fix2023-08-18T11:13:51Zharshit aggarwalAzure - pipeline fix## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M9 - Release 0.12harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/76GONRG-3109: Move common logic to osdu-airflow-lib2021-09-13T08:38:35ZAleksandr Spivakov (EPAM)GONRG-3109: Move common logic to osdu-airflow-lib## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X]...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X] IBM
## Updates description?
Move common logic to osdu-airflow-lib
Closes #89M9 - Release 0.12Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/75GONRG-3107: Change backward compatibility2021-09-10T15:14:21ZYan Sushchynski (EPAM)GONRG-3107: Change backward compatibility## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Updates description?
The DAG depends on [osdu-airflow-lib](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib) now.
Before deploying the DAG install the package
```shell
pip install 'osdu-airflow' --extra-index-url=https://community.opengroup.org/api/v4/projects/668/packages/pypi/simple
```M9 - Release 0.12Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/74Azure - Update ADO pipeline2023-08-18T11:13:52Zharshit aggarwalAzure - Update ADO pipeline## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M9 - Release 0.12harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/73Azure - Update Test Dag stage2023-08-18T11:13:54Zharshit aggarwalAzure - Update Test Dag stage## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M9 - Release 0.12harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/100Revert "Merge branch 'cherry-pick-66f449e1' into 'release/0.13'"2022-02-10T08:52:47ZDavid Diederichd.diederich@opengroup.orgRevert "Merge branch 'cherry-pick-66f449e1' into 'release/0.13'"## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/98Cherry-pick 'Adding extra index urls to enable finding these packages' into '...2022-02-09T18:02:47ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Adding extra index urls to enable finding these packages' into 'release/0.13'Original MR: !96Original MR: !96M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/96Adding extra index urls to enable finding these packages2022-02-09T17:16:12ZDavid Diederichd.diederich@opengroup.orgAdding extra index urls to enable finding these packages## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IB...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Adding in the --extra-index-url options to the `requirements.txt` file so this can build out of the boxM10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/95Azure - Add zip file2023-08-18T11:13:35Zharshit aggarwalAzure - Add zip fileM10 - Release 0.13harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/93Delete osdu-ingest-r2.py2023-08-18T11:13:37ZYan Sushchynski (EPAM)Delete osdu-ingest-r2.py## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Delete deprecated DAGM10 - Release 0.13Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/92[GONRG-4086] Use PyPi package version from master2023-08-18T11:13:39ZDanylo Vanin (EPAM)[GONRG-4086] Use PyPi package version from master## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] I...## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description
Changed bootstrap script for GCP to generally use master version of PyPi packages (ex. 0.13.0.dev1).
For the release of current service - versions of PyPi packages must be manually set to use release version -> ex. 0.12.0
PyPi packages versions are set in `./devops/gcp/composer_requirements.txt`M10 - Release 0.13Danylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/91GONRG-4002: Increase DAG timeout to process large Manifests2023-08-18T11:13:41ZYan Sushchynski (EPAM)GONRG-4002: Increase DAG timeout to process large Manifests## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Sometime the DAG fails on processing 50k records Manifests. It turned out that the DAG has a timeout set to 1 hour. If we increase this value, we are able to process such large ManifestsM10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/90GONRG-3893-bootstrap: added new variables: sa_file_path, data_partition,...2023-08-18T11:13:42ZAliaksandr Lubouski (EPAM)GONRG-3893-bootstrap: added new variables: sa_file_path, data_partition,...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [X] Bootstrap
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [X] Bootstrap
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/89GONRG-3943: added error catch for no config change2023-08-18T11:13:44ZDanylo Vanin (EPAM)GONRG-3943: added error catch for no config change## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description
Added catching error in bash script for the case when airflow pypi packages configuration is up-to-date.M10 - Release 0.13Danylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/88GONRG-3779: Common pipeline for dag2021-12-22T18:57:02ZYan Sushchynski (EPAM)GONRG-3779: Common pipeline for dag## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Create common pipelines for GCP.M10 - Release 0.13Vladislav Shishko (EPAM)Vladislav Shishko (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/87GONRG-3675-bootstrap: added pipeline, Dockerfile, as well as configuration...2023-08-18T11:13:47ZAliaksandr Lubouski (EPAM)GONRG-3675-bootstrap: added pipeline, Dockerfile, as well as configuration...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [x] Deployment Framework Integration
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [x] Deployment Framework Integration
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
GONRG-3675-bootstrap: added pipeline, Dockerfile, as well as configuration files with bootstrap bash scriptM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/85GONRG-3733 remove airflow v1 deployment steps2021-11-11T09:57:52ZAleksandr Spivakov (EPAM)GONRG-3733 remove airflow v1 deployment steps## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M10 - Release 0.13Siarhei Khaletski (EPAM)Yan Sushchynski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/84GONRG-3709 fix E2E tests2021-11-03T08:46:09ZAleksandr Spivakov (EPAM)GONRG-3709 fix E2E tests## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
fix E2E tests for GCP:
- add support for tags in CI steps
- update required services URLs for Community test environmentM10 - Release 0.13Siarhei Khaletski (EPAM)Yan Sushchynski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/101Update bootstrap names [GONRG-4580]2023-08-18T11:13:31ZMikhail Piatliou (EPAM)Update bootstrap names [GONRG-4580]M11 - Release 0.14Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)