Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2023-01-25T14:26:21Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/140Draft: Temporarily retrieve job artifact for checking [GONRG-6280]2023-01-25T14:26:21ZMikhail Piatliou (EPAM)Draft: Temporarily retrieve job artifact for checking [GONRG-6280]## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBMMikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/136Configure retry for OSDU Ingest2023-01-13T05:57:16ZSrishti SharmaConfigure retry for OSDU Ingest**PBI:** [https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\\\_workitems/edit/15576](https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/%5C_workitems/edit/15576) Setting up retries for OSDU ingest
=======...**PBI:** [https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\\\_workitems/edit/15576](https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/%5C_workitems/edit/15576) Setting up retries for OSDU ingest
============== Tested the retries on a test instance and verified the functionality is working. Test Dags:
* https://adf-4q2m464plpuwq-airflowir-2707.westeurope.airflow.svc.datafactory.azure.com/tree?dag_id=sstest_Osdu_ingest
* https://adf-4q2m464plpuwq-airflowir-2707.westeurope.airflow.svc.datafactory.azure.com/tree?dag_id=sstest_csv-parser
* https://adf-4q2m464plpuwq-airflowir-2707.westeurope.airflow.svc.datafactory.azure.com/log?dag_id=sstest0301_osdu-ingest-r3&task_id=process_manifest_task_1&execution_date=2023-01-03T08%3A37%3A16.035579%2B00%3A00
=====================================
Template -
original osdu-ingest-r3 file:
![image](/uploads/62601ad1398db179169514800efea5b1/image.png)
Generated osdu-ingest-r3 file after template replaces placeholders:
![image](/uploads/92e8649281d981f36df0c2e4c1dcab9b/image.png)
Tested in airflow portal and retries were successful:
![image](/uploads/59e4adb264b129b964b3094c1e1749f1/image.png)
https://adf-4q2m464plpuwq-airflowir-2707.westeurope.airflow.svc.datafactory.azure.com/log?dag_id=sstest0301_osdu-ingest-r3&task_id=process_manifest_task_1&execution_date=2023-01-03T08%3A37%3A16.035579%2B00%3A00https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/138Merge branch 'GONRG-6172-Fix-Postman-Collection-Url' into 'master'2022-12-21T17:30:29ZDanylo Vanin (EPAM)Merge branch 'GONRG-6172-Fix-Postman-Collection-Url' into 'master'## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
Cherry picked from https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/137M15 - Release 0.18Oleksandr Kosse (EPAM)Danylo Vanin (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/135Configured retrues in osdu-ingest2022-12-17T11:11:19ZSrishti SharmaConfigured retrues in osdu-ingest**PBI:** https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_workitems/edit/15576 Setting up retries for OSDU ingest
============== Tested the retries on a test instance and verified the functionality is working. Test Da...**PBI:** https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_workitems/edit/15576 Setting up retries for OSDU ingest
============== Tested the retries on a test instance and verified the functionality is working. Test Dags:
* https://adf-4q2m464plpuwq-airflowir-2707.westeurope.airflow.svc.datafactory.azure.com/tree?dag_id=sstest_Osdu_ingest
* https://adf-4q2m464plpuwq-airflowir-2707.westeurope.airflow.svc.datafactory.azure.com/tree?dag_id=sstest_csv-parserhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/125Merge ibm helm to master2022-12-12T21:30:13ZManish SinghMerge ibm helm to master## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)
Merge ibm helm to masterM14 - Release 0.17Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/134Manifest by reference - DAG add2022-12-02T14:27:27Zshivani karipeManifest by reference - DAG add## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Adding the DAG from manifest-by-reference branch to Master.M15 - Release 0.18Valentin GauthierNaveen RamachandraiahValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/64GONRG-2913: Added support for whitelist reference patterns to exclude from...2022-12-01T11:24:31ZAleksandr Spivakov (EPAM)GONRG-2913: Added support for whitelist reference patterns to exclude from...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Introduces availability to whitelist references using custom regexp patterns to exclude those references from referential integrity validation.
New Airflow variable example:
![image](/uploads/771f4da7bde79cfef6d228777ff5dd7d/image.png)
See related MR for more details:
https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/22
closes https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/external-data-framework/-/issues/180M8 - Release 0.11Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/133Manifest by reference2022-11-30T09:19:51ZValentin GauthierManifest by reference## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M15 - Release 0.18Chad LeongValentin GauthierChad Leonghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/132Add osdu-ingest-r3-by-reference DAG2022-11-29T14:47:16ZHamdaan KhalidAdd osdu-ingest-r3-by-reference DAG## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
This PR adds a the new dag osdu-ingest-r3-by-reference into this repoitory.M15 - Release 0.18Hamdaan KhalidHamdaan Khalidhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/131M14 Upgrade2022-11-20T11:52:15ZAnkur RawatM14 UpgradeM12 changes into M14M12 changes into M14Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/130M14 Upgrade2022-11-16T13:06:38ZAnkur RawatM14 UpgradeM12 changes into M14M12 changes into M14Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/128added changes to fix azure tests2022-10-27T05:15:20Zshivani karipeadded changes to fix azure tests## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [*] Azure
- [ ] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [*] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M15 - Release 0.18shivani karipeshivani karipehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/123GONRG-5659: Add versioning to DAGs2022-10-24T10:16:28ZYan Sushchynski (EPAM)GONRG-5659: Add versioning to DAGs## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Add versioning for GCP DAGsM15 - Release 0.18Yan Sushchynski (EPAM)Yan Sushchynski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/127DO NOT MERGE2022-10-17T12:01:30Zshivani karipeDO NOT MERGE## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/126Merge branch 'merge-ibm-helm' into 'master'2022-10-12T04:03:06ZShrikant GargMerge branch 'merge-ibm-helm' into 'master'Original MR: !125Original MR: !125M14 - Release 0.17David Diederichd.diederich@opengroup.orgChad LeongDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/122GONRG-5659: Add versioning to DAGs2022-09-26T12:38:17ZYan Sushchynski (EPAM)GONRG-5659: Add versioning to DAGs## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M15 - Release 0.18https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/118Revert "Merge branch 'fix-requirements-file' into 'master'"2022-07-29T09:12:01ZVivek OjhaRevert "Merge branch 'fix-requirements-file' into 'master'"## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Vivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/120Pipeline for dags in dev2 and preship [GONRG-5150]2022-07-28T12:26:30ZMikhail Piatliou (EPAM)Pipeline for dags in dev2 and preship [GONRG-5150]M13 - Release 0.16Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/119GCP common pipeline dags [GONRG-5296]2022-07-26T10:04:29ZMikhail Piatliou (EPAM)GCP common pipeline dags [GONRG-5296]M13 - Release 0.16Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/116Draft: GONRG-5257: cherry-pick of dags pipeline2022-06-30T09:34:50ZYauheni Rykhter (EPAM)Draft: GONRG-5257: cherry-pick of dags pipeline## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)