Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2023-08-18T11:13:32Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/99Revert "Merge branch 'fix-requirements-file' into 'master'"2023-08-18T11:13:32ZDavid Diederichd.diederich@opengroup.orgRevert "Merge branch 'fix-requirements-file' into 'master'"## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M11 - Release 0.14David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/97GONRG-4139 Split dags bootstrap into composer provisioning and DAGs installation2023-08-18T11:13:34ZMaksimelyan Tamashevich (EPAM)GONRG-4139 Split dags bootstrap into composer provisioning and DAGs installation## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Split dags bootstrap job into composer provisioning and DAGs installation jobsM11 - Release 0.14Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/94GONRG-4004: Replace bootstrap values with AF vars2022-02-28T15:01:08ZYan Sushchynski (EPAM)GONRG-4004: Replace bootstrap values with AF vars## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Change bootstrap values to AF variablesM11 - Release 0.14Aleksandr Spivakov (EPAM)Aleksandr Spivakov (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/111Ibm pipeline2022-05-13T11:35:15ZWalter DIbm pipeline## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] I...## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] IBM
## Updates description?
Added CICD pipeline for IBM.M12 - Release 0.15Walter DWalter Dhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/110Move airflow anthos containerize jobs to infra repo [GONRG-4872]2023-08-18T11:13:22ZMikhail Piatliou (EPAM)Move airflow anthos containerize jobs to infra repo [GONRG-4872]M12 - Release 0.15Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/108[GONRG-4699] Refactor bootstrap pipeline2023-08-18T11:13:24ZDanylo Vanin (EPAM)[GONRG-4699] Refactor bootstrap pipeline## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M12 - Release 0.15Danylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/107Anthos bootstrap skip2022-04-21T12:36:12ZYan Sushchynski (EPAM)Anthos bootstrap skip## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Add skipp of Anthos bootstrap step and publish DAGs.M12 - Release 0.15https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/106Tags for gcp jobs [GONRG-4672]2023-08-18T11:13:26ZMikhail Piatliou (EPAM)Tags for gcp jobs [GONRG-4672]M12 - Release 0.15Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/105add all files for task airflow scheduler2023-08-18T11:13:27ZAliaksei Kruk2add all files for task airflow schedulerM12 - Release 0.15Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/104Gonrg 4571 add pipeline af worker2023-08-18T11:13:29ZAliaksei Kruk2Gonrg 4571 add pipeline af workerM12 - Release 0.15Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/120Pipeline for dags in dev2 and preship [GONRG-5150]2022-07-28T12:26:30ZMikhail Piatliou (EPAM)Pipeline for dags in dev2 and preship [GONRG-5150]M13 - Release 0.16Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/119GCP common pipeline dags [GONRG-5296]2022-07-26T10:04:29ZMikhail Piatliou (EPAM)GCP common pipeline dags [GONRG-5296]M13 - Release 0.16Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/117Gonrg 5216 create deploy da gs on composer for anthos2023-08-18T11:13:17ZMaksimelyan Tamashevich (EPAM)Gonrg 5216 create deploy da gs on composer for anthos## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M13 - Release 0.16Oleksandr Kosse (EPAM)Yan Sushchynski (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/115Gonrg 5061 refactor comm pipe2023-08-18T11:13:19ZSiarhei Symanovich (EPAM)Gonrg 5061 refactor comm pipe## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M13 - Release 0.16Siarhei Symanovich (EPAM)Siarhei Symanovich (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/114Manifest ingestion e2e tests2023-08-18T11:13:21ZScott HowardManifest ingestion e2e tests## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)
Added AWS variables and includes for E2E testing.M13 - Release 0.16Scott HowardScott Howardhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/112Gonrg 4996 anthos dags int tests2022-06-20T14:25:24ZYan Sushchynski (EPAM)Gonrg 4996 anthos dags int tests## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Add tnhos steps to the pipelineM13 - Release 0.16https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/126Merge branch 'merge-ibm-helm' into 'master'2022-10-12T04:03:06ZShrikant GargMerge branch 'merge-ibm-helm' into 'master'Original MR: !125Original MR: !125M14 - Release 0.17David Diederichd.diederich@opengroup.orgChad LeongDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/125Merge ibm helm to master2022-12-12T21:30:13ZManish SinghMerge ibm helm to master## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)
Merge ibm helm to masterM14 - Release 0.17Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/121GONRG-5398-cleanup-dag-docker-images2023-08-18T11:13:16ZTejinder DhingraGONRG-5398-cleanup-dag-docker-images## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
override-stages.yml: Disabled and ommitted osdu-gcp-containerize-bootstrap-* jobsM14 - Release 0.17https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/138Merge branch 'GONRG-6172-Fix-Postman-Collection-Url' into 'master'2022-12-21T17:30:29ZDanylo Vanin (EPAM)Merge branch 'GONRG-6172-Fix-Postman-Collection-Url' into 'master'## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
Cherry picked from https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/137M15 - Release 0.18Oleksandr Kosse (EPAM)Danylo Vanin (EPAM)Oleksandr Kosse (EPAM)