Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2023-09-01T11:32:45Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/155Add gc_manifest_version2023-09-01T11:32:45ZYan Sushchynski (EPAM)Add gc_manifest_version## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/154fix pipeline2023-08-04T11:58:44Zsaketh somarajufix pipeline## Type of change
- [x] Pipeline fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
Fix azure tag pipeline## Type of change
- [x] Pipeline fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
Fix azure tag pipelineM19 - Release 0.22saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/144Draft: GONRG-6642: update release image for dag2023-03-02T09:34:19ZYauheni Rykhter (EPAM)Draft: GONRG-6642: update release image for dag## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GC
- [ ] I...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GC
- [ ] IBM
## Updates description?
Update image name for dagYauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/141Draft: Fix DAGs deploy jobs [GONRG-6380]2023-01-27T12:59:33ZMikhail Piatliou (EPAM)Draft: Fix DAGs deploy jobs [GONRG-6380]## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBMMikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/140Draft: Temporarily retrieve job artifact for checking [GONRG-6280]2023-01-25T14:26:21ZMikhail Piatliou (EPAM)Draft: Temporarily retrieve job artifact for checking [GONRG-6280]## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBMMikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/139Configure retry for OSDU Ingest2023-08-25T22:21:27ZSrishti SharmaConfigure retry for OSDU IngestConfigured Operator level retries OSDU ingest
Tested the retries on a test instance and verified the functionality is working.Configured Operator level retries OSDU ingest
Tested the retries on a test instance and verified the functionality is working.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/135Configured retrues in osdu-ingest2022-12-17T11:11:19ZSrishti SharmaConfigured retrues in osdu-ingest**PBI:** https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_workitems/edit/15576 Setting up retries for OSDU ingest
============== Tested the retries on a test instance and verified the functionality is working. Test Da...**PBI:** https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_workitems/edit/15576 Setting up retries for OSDU ingest
============== Tested the retries on a test instance and verified the functionality is working. Test Dags:
* https://adf-4q2m464plpuwq-airflowir-2707.westeurope.airflow.svc.datafactory.azure.com/tree?dag_id=sstest_Osdu_ingest
* https://adf-4q2m464plpuwq-airflowir-2707.westeurope.airflow.svc.datafactory.azure.com/tree?dag_id=sstest_csv-parserhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/133Manifest by reference2022-11-30T09:19:51ZValentin GauthierManifest by reference## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M15 - Release 0.18Chad LeongValentin GauthierChad Leonghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/132Add osdu-ingest-r3-by-reference DAG2022-11-29T14:47:16ZHamdaan KhalidAdd osdu-ingest-r3-by-reference DAG## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
This PR adds a the new dag osdu-ingest-r3-by-reference into this repoitory.M15 - Release 0.18Hamdaan KhalidHamdaan Khalidhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/130M14 Upgrade2022-11-16T13:06:38ZAnkur RawatM14 UpgradeM12 changes into M14M12 changes into M14Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/128added changes to fix azure tests2022-10-27T05:15:20Zshivani karipeadded changes to fix azure tests## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [*] Azure
- [ ] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [*] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M15 - Release 0.18shivani karipeshivani karipehttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/127DO NOT MERGE2022-10-17T12:01:30Zshivani karipeDO NOT MERGE## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/122GONRG-5659: Add versioning to DAGs2022-09-26T12:38:17ZYan Sushchynski (EPAM)GONRG-5659: Add versioning to DAGs## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M15 - Release 0.18https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/116Draft: GONRG-5257: cherry-pick of dags pipeline2022-06-30T09:34:50ZYauheni Rykhter (EPAM)Draft: GONRG-5257: cherry-pick of dags pipeline## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/113E2E Testing for Manifest Ingestion2022-05-31T16:31:37ZScott HowardE2E Testing for Manifest Ingestion## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Scott HowardScott Howardhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/109update dir for images2022-04-27T11:55:41ZAliaksei Kruk2update dir for images## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/103add a pipeline build image airflow-worker2022-04-05T08:35:11ZAliaksei Kruk2add a pipeline build image airflow-workerOleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/102Upgrade First Party Library Dependencies for Release 0.142022-04-05T10:40:28ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.14This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: bfdf1bdf75859c5d33a7a43641b75a6b128d2472
Python:
```
| Python Dependencies | devops/gcp/composer_requirements.txt | src/osdu_dags/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------ |
| osdu-airflow | 0.12.0 | 0.12.0 |
| osdu-api | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.1 | 0.12.1 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: afbf3fc98ab707e3328071a883e020d087eab91f
Python:
```
| Python Dependencies | devops/gcp/composer_requirements.txt | src/osdu_dags/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------ |
| osdu-airflow | 0.12.0 | 0.12.0 |
| osdu-api | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.1 | 0.12.1 |M11 - Release 0.14https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/72GONRG-2921: Batch processing records2021-09-09T11:39:40ZYan Sushchynski (EPAM)GONRG-2921: Batch processing records## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Add possibility to turn on Save records by batches on Storage Service.M9 - Release 0.12Siarhei Khaletski (EPAM)Yan Sushchynski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/45Cherry-pick "Ingestion updates" into the release branch2021-05-15T00:56:16ZDavid Diederichd.diederich@opengroup.orgCherry-pick "Ingestion updates" into the release branchOriginal MR: osdu/platform/data-flow/ingestion/ingestion-dags!44
Also added some essential code from: osdu/platform/data-flow/ingestion/ingestion-dags!38Original MR: osdu/platform/data-flow/ingestion/ingestion-dags!44
Also added some essential code from: osdu/platform/data-flow/ingestion/ingestion-dags!38David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.org