Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2020-08-25T12:35:02Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/2WIP: Merge Energistics parser code into DAG repo (GONRG-572)2020-08-25T12:35:02ZDmitriy RudkoWIP: Merge Energistics parser code into DAG repo (GONRG-572)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/8Fixed import errors with latest sdk2021-02-03T14:55:51ZKishore BattulaFixed import errors with latest sdkhttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/2https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/2Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/23Draft: Merge Release/0.5 into Master2021-02-25T10:46:51ZSiarhei Khaletski (EPAM)Draft: Merge Release/0.5 into Master## Type of change
- [ ] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
This MR contains hot-fixes that were done during release branch supportSiarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/25Draft: Fix id regexps in referential integrity validation2021-03-03T09:27:00ZYan Sushchynski (EPAM)Draft: Fix id regexps in referential integrity validation## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Fixed bug when ID regular expressions could have match kinds and other values in referential integrity validation by making these regular expressions more strict.Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/33release 0.7 hotfix - master data ingestion fails2021-03-23T20:01:08ZDmitriy Rudkorelease 0.7 hotfix - master data ingestion fails## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/43WIP: Add report of skipped ids2021-04-19T15:14:58ZYan Sushchynski (EPAM)WIP: Add report of skipped ids## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Add report about skipped and processed ids to XComs (https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/35).Siarhei Khaletski (EPAM)Rostislav Dublin (EPAM)Michael Tarasov (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/41GONRG-2147: Fix double colon in refs2021-04-19T15:15:11ZYan Sushchynski (EPAM)GONRG-2147: Fix double colon in refs## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
These changes fix the issue, when references to ingested entities get extra ":"
(e.g. `"Datasets": ["osdu:dataset--File.Generic:feb02::"]` instead of `"Datasets": [ "osdu:dataset--File.Generic:feb02:"]`).Siarhei Khaletski (EPAM)Rostislav Dublin (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/42Bugfix/gonrg 2144 id integer part considered version2021-04-19T15:15:35ZYan Sushchynski (EPAM)Bugfix/gonrg 2144 id integer part considered version## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Fix the issue, when an integer part of ids was considered a version and this prevented WP manifest ingestion.
(https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/55)Siarhei Khaletski (EPAM)Rostislav Dublin (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/38Bugfix/remove duplicate steps2021-04-22T19:44:21ZSiarhei Khaletski (EPAM)Bugfix/remove duplicate steps@divido merge please (already merged into `master`)@divido merge please (already merged into `master`)David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/45Cherry-pick "Ingestion updates" into the release branch2021-05-15T00:56:16ZDavid Diederichd.diederich@opengroup.orgCherry-pick "Ingestion updates" into the release branchOriginal MR: osdu/platform/data-flow/ingestion/ingestion-dags!44
Also added some essential code from: osdu/platform/data-flow/ingestion/ingestion-dags!38Original MR: osdu/platform/data-flow/ingestion/ingestion-dags!44
Also added some essential code from: osdu/platform/data-flow/ingestion/ingestion-dags!38David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/35Increased search result limt2021-05-27T14:34:02ZKishore BattulaIncreased search result limt## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Increasing the search limit to 200 as by default only 10 results are returned from search service. The correct implementation is call search service multiple times to fetch all the results in the system. This is a quick fix to unblock clients testing manifest ingestionhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/39removing with_validation boolean flag2021-07-05T11:53:01ZBrady Spiva [AWS]removing with_validation boolean flag## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
This addresses issue #54, removing redundant validation steps to speed up manifest parsing. The manifest parser DAG performs validations for referential integrity and schema conformity in separate operators, so there is no need to repeat the validations here in the manifest parsing stage of this DAG.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/72GONRG-2921: Batch processing records2021-09-09T11:39:40ZYan Sushchynski (EPAM)GONRG-2921: Batch processing records## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Add possibility to turn on Save records by batches on Storage Service.M9 - Release 0.12Siarhei Khaletski (EPAM)Yan Sushchynski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/103add a pipeline build image airflow-worker2022-04-05T08:35:11ZAliaksei Kruk2add a pipeline build image airflow-workerOleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/102Upgrade First Party Library Dependencies for Release 0.142022-04-05T10:40:28ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.14This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: bfdf1bdf75859c5d33a7a43641b75a6b128d2472
Python:
```
| Python Dependencies | devops/gcp/composer_requirements.txt | src/osdu_dags/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------ |
| osdu-airflow | 0.12.0 | 0.12.0 |
| osdu-api | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.1 | 0.12.1 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: afbf3fc98ab707e3328071a883e020d087eab91f
Python:
```
| Python Dependencies | devops/gcp/composer_requirements.txt | src/osdu_dags/requirements.txt |
| ------------------- | ------------------------------------ | ------------------------------ |
| osdu-airflow | 0.12.0 | 0.12.0 |
| osdu-api | 0.12.1 | 0.12.1 |
| osdu-ingestion | 0.12.1 | 0.12.1 |M11 - Release 0.14https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/109update dir for images2022-04-27T11:55:41ZAliaksei Kruk2update dir for images## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/113E2E Testing for Manifest Ingestion2022-05-31T16:31:37ZScott HowardE2E Testing for Manifest Ingestion## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] ...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Scott HowardScott Howardhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/116Draft: GONRG-5257: cherry-pick of dags pipeline2022-06-30T09:34:50ZYauheni Rykhter (EPAM)Draft: GONRG-5257: cherry-pick of dags pipeline## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/122GONRG-5659: Add versioning to DAGs2022-09-26T12:38:17ZYan Sushchynski (EPAM)GONRG-5659: Add versioning to DAGs## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)M15 - Release 0.18https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/127DO NOT MERGE2022-10-17T12:01:30Zshivani karipeDO NOT MERGE## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)