Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2022-09-26T12:38:17Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/122GONRG-5659: Add versioning to DAGs2022-09-26T12:38:17ZYan Sushchynski (EPAM)GONRG-5659: Add versioning to DAGs## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M15 - Release 0.18https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/121GONRG-5398-cleanup-dag-docker-images2023-08-18T11:13:16ZTejinder DhingraGONRG-5398-cleanup-dag-docker-images## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
override-stages.yml: Disabled and ommitted osdu-gcp-containerize-bootstrap-* jobsM14 - Release 0.17https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/120Pipeline for dags in dev2 and preship [GONRG-5150]2022-07-28T12:26:30ZMikhail Piatliou (EPAM)Pipeline for dags in dev2 and preship [GONRG-5150]M13 - Release 0.16Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/119GCP common pipeline dags [GONRG-5296]2022-07-26T10:04:29ZMikhail Piatliou (EPAM)GCP common pipeline dags [GONRG-5296]M13 - Release 0.16Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/118Revert "Merge branch 'fix-requirements-file' into 'master'"2022-07-29T09:12:01ZVivek OjhaRevert "Merge branch 'fix-requirements-file' into 'master'"## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Vivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/117Gonrg 5216 create deploy da gs on composer for anthos2023-08-18T11:13:17ZMaksimelyan Tamashevich (EPAM)Gonrg 5216 create deploy da gs on composer for anthos## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M13 - Release 0.16Oleksandr Kosse (EPAM)Yan Sushchynski (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/116Draft: GONRG-5257: cherry-pick of dags pipeline2022-06-30T09:34:50ZYauheni Rykhter (EPAM)Draft: GONRG-5257: cherry-pick of dags pipeline## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/115Gonrg 5061 refactor comm pipe2023-08-18T11:13:19ZSiarhei Symanovich (EPAM)Gonrg 5061 refactor comm pipe## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M13 - Release 0.16Siarhei Symanovich (EPAM)Siarhei Symanovich (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/114Manifest ingestion e2e tests2023-08-18T11:13:21ZScott HowardManifest ingestion e2e tests## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)
Added AWS variables and includes for E2E testing.M13 - Release 0.16Scott HowardScott Howardhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/113E2E Testing for Manifest Ingestion2022-05-31T16:31:37ZScott HowardE2E Testing for Manifest Ingestion## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Scott HowardScott Howardhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/112Gonrg 4996 anthos dags int tests2022-06-20T14:25:24ZYan Sushchynski (EPAM)Gonrg 4996 anthos dags int tests## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Add tnhos steps to the pipelineM13 - Release 0.16https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/111Ibm pipeline2022-05-13T11:35:15ZWalter DIbm pipeline## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] I...## Type of change
- [] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] IBM
## Updates description?
Added CICD pipeline for IBM.M12 - Release 0.15Walter DWalter Dhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/110Move airflow anthos containerize jobs to infra repo [GONRG-4872]2023-08-18T11:13:22ZMikhail Piatliou (EPAM)Move airflow anthos containerize jobs to infra repo [GONRG-4872]M12 - Release 0.15Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/109update dir for images2022-04-27T11:55:41ZAliaksei Kruk2update dir for images## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/107Anthos bootstrap skip2022-04-21T12:36:12ZYan Sushchynski (EPAM)Anthos bootstrap skip## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Add skipp of Anthos bootstrap step and publish DAGs.M12 - Release 0.15https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/106Tags for gcp jobs [GONRG-4672]2023-08-18T11:13:26ZMikhail Piatliou (EPAM)Tags for gcp jobs [GONRG-4672]M12 - Release 0.15Mikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/105add all files for task airflow scheduler2023-08-18T11:13:27ZAliaksei Kruk2add all files for task airflow schedulerM12 - Release 0.15Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/104Gonrg 4571 add pipeline af worker2023-08-18T11:13:29ZAliaksei Kruk2Gonrg 4571 add pipeline af workerM12 - Release 0.15Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/103add a pipeline build image airflow-worker2022-04-05T08:35:11ZAliaksei Kruk2add a pipeline build image airflow-workerOleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/102Upgrade First Party Library Dependencies for Release 0.142022-04-05T10:40:28ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.14This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: bfdf1bdf75859c5d33a7a43641b75a6b128d2472
Python:
```
| Python Dependencies | devops/gcp/composer_requirements.txt | src/osdu_dags/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------ |
| osdu-airflow | 0.12.0 | 0.12.0 |
| osdu-api | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.1 | 0.12.1 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: afbf3fc98ab707e3328071a883e020d087eab91f
Python:
```
| Python Dependencies | devops/gcp/composer_requirements.txt | src/osdu_dags/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------ |
| osdu-airflow | 0.12.0 | 0.12.0 |
| osdu-api | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.1 | 0.12.1 |M11 - Release 0.14