Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2023-08-18T11:14:40Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/30Refactor referential integrity (GONRG-1932)2023-08-18T11:14:40ZSiarhei Khaletski (EPAM)Refactor referential integrity (GONRG-1932)## Type of change
- [ ] Bug Fix
- [ ] Feature
- [x] Refactoring
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azur...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [x] Refactoring
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
- Manifest integrity ensuring logic has been refactored.M5 - Release 0.8Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/32Fix for FileSource; fix for pipelines with invalid manifest2023-08-18T11:14:39ZSiarhei Khaletski (EPAM)Fix for FileSource; fix for pipelines with invalid manifest## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
- Fix for issue with FileSource as a string with spaces
- Fix for with failing pipeline for generally invalid manifestM5 - Release 0.8Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/34Added BaseTokenRefresher2023-08-18T11:14:37ZSiarhei Khaletski (EPAM)Added BaseTokenRefresher## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
- Added BaseTokenRefresherM5 - Release 0.8Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/36GONRG-2008: Documentation has been updated2023-08-18T11:14:36ZSiarhei Khaletski (EPAM)GONRG-2008: Documentation has been updated## Type of change
- [ ] Bug Fix
- [ ] Feature
- [x] Documentation
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Az...## Type of change
- [ ] Bug Fix
- [ ] Feature
- [x] Documentation
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
- Documentation has been updatedM5 - Release 0.8Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/38Bugfix/remove duplicate steps2021-04-22T19:44:21ZSiarhei Khaletski (EPAM)Bugfix/remove duplicate steps@divido merge please (already merged into `master`)@divido merge please (already merged into `master`)David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/44Ingestion updates2023-08-18T11:14:31ZYan Sushchynski (EPAM)Ingestion updates## Type of change
- [x] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
This MR comes with a few updates:
Features:
* Add report about skipped and processed ids to XComs (Issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/35). (GONRG-1934)
Bugfixes:
* Fix the issue, when an integer part of ids was considered a version and this prevented WP manifest ingestion. (Isue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/55). (GONRG-2144)
* Fix the issue, when references to ingested entities with real ids get extra ":" (e.g. `"Datasets": ["osdu:dataset--File.Generic:feb02::"]` instead of `"Datasets": [ "osdu:dataset--File.Generic:feb02:"]`) (Issue: https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/142). (GONRG-2147)M5 - Release 0.8Siarhei Khaletski (EPAM)Rostislav Dublin (EPAM)Siarhei Khaletski (EPAM)