Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2022-04-05T10:40:28Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/102Upgrade First Party Library Dependencies for Release 0.142022-04-05T10:40:28ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.14This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: bfdf1bdf75859c5d33a7a43641b75a6b128d2472
Python:
```
| Python Dependencies | devops/gcp/composer_requirements.txt | src/osdu_dags/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------ |
| osdu-airflow | 0.12.0 | 0.12.0 |
| osdu-api | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.1 | 0.12.1 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: afbf3fc98ab707e3328071a883e020d087eab91f
Python:
```
| Python Dependencies | devops/gcp/composer_requirements.txt | src/osdu_dags/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------ |
| osdu-airflow | 0.12.0 | 0.12.0 |
| osdu-api | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.1 | 0.12.1 |M11 - Release 0.14https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/101Update bootstrap names [GONRG-4580]2023-08-18T11:13:31ZMikhail Piatliou (EPAM)Update bootstrap names [GONRG-4580]M11 - Release 0.14Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/99Revert "Merge branch 'fix-requirements-file' into 'master'"2023-08-18T11:13:32ZDavid Diederichd.diederich@opengroup.orgRevert "Merge branch 'fix-requirements-file' into 'master'"## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M11 - Release 0.14David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/97GONRG-4139 Split dags bootstrap into composer provisioning and DAGs installation2023-08-18T11:13:34ZMaksimelyan Tamashevich (EPAM)GONRG-4139 Split dags bootstrap into composer provisioning and DAGs installation## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Split dags bootstrap job into composer provisioning and DAGs installation jobsM11 - Release 0.14Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/94GONRG-4004: Replace bootstrap values with AF vars2022-02-28T15:01:08ZYan Sushchynski (EPAM)GONRG-4004: Replace bootstrap values with AF vars## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Change bootstrap values to AF variablesM11 - Release 0.14Aleksandr Spivakov (EPAM)Aleksandr Spivakov (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/111Ibm pipeline2022-05-13T11:35:15ZWalter DIbm pipeline## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] I...## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] IBM
## Updates description?
Added CICD pipeline for IBM.M12 - Release 0.15Walter DWalter Dhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/110Move airflow anthos containerize jobs to infra repo [GONRG-4872]2023-08-18T11:13:22ZMikhail Piatliou (EPAM)Move airflow anthos containerize jobs to infra repo [GONRG-4872]M12 - Release 0.15Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/108[GONRG-4699] Refactor bootstrap pipeline2023-08-18T11:13:24ZDanylo Vanin (EPAM)[GONRG-4699] Refactor bootstrap pipeline## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M12 - Release 0.15Danylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/107Anthos bootstrap skip2022-04-21T12:36:12ZYan Sushchynski (EPAM)Anthos bootstrap skip## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Add skipp of Anthos bootstrap step and publish DAGs.M12 - Release 0.15https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/106Tags for gcp jobs [GONRG-4672]2023-08-18T11:13:26ZMikhail Piatliou (EPAM)Tags for gcp jobs [GONRG-4672]M12 - Release 0.15Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/105add all files for task airflow scheduler2023-08-18T11:13:27ZAliaksei Kruk2add all files for task airflow schedulerM12 - Release 0.15Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/104Gonrg 4571 add pipeline af worker2023-08-18T11:13:29ZAliaksei Kruk2Gonrg 4571 add pipeline af workerM12 - Release 0.15Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/120Pipeline for dags in dev2 and preship [GONRG-5150]2022-07-28T12:26:30ZMikhail Piatliou (EPAM)Pipeline for dags in dev2 and preship [GONRG-5150]M13 - Release 0.16Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/119GCP common pipeline dags [GONRG-5296]2022-07-26T10:04:29ZMikhail Piatliou (EPAM)GCP common pipeline dags [GONRG-5296]M13 - Release 0.16Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/117Gonrg 5216 create deploy da gs on composer for anthos2023-08-18T11:13:17ZMaksimelyan Tamashevich (EPAM)Gonrg 5216 create deploy da gs on composer for anthos## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M13 - Release 0.16Oleksandr Kosse (EPAM)Yan Sushchynski (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/115Gonrg 5061 refactor comm pipe2023-08-18T11:13:19ZSiarhei Symanovich (EPAM)Gonrg 5061 refactor comm pipe## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M13 - Release 0.16Siarhei Symanovich (EPAM)Siarhei Symanovich (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/114Manifest ingestion e2e tests2023-08-18T11:13:21ZScott HowardManifest ingestion e2e tests## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)
Added AWS variables and includes for E2E testing.M13 - Release 0.16Scott HowardScott Howardhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/112Gonrg 4996 anthos dags int tests2022-06-20T14:25:24ZYan Sushchynski (EPAM)Gonrg 4996 anthos dags int tests## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Add tnhos steps to the pipelineM13 - Release 0.16https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/126Merge branch 'merge-ibm-helm' into 'master'2022-10-12T04:03:06ZShrikant GargMerge branch 'merge-ibm-helm' into 'master'Original MR: !125Original MR: !125M14 - Release 0.17David Diederichd.diederich@opengroup.orgChad LeongDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/125Merge ibm helm to master2022-12-12T21:30:13ZManish SinghMerge ibm helm to master## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)
Merge ibm helm to masterM14 - Release 0.17Anuj GuptaAnuj Gupta