Manifest Ingestion DAG merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests2023-08-18T11:15:09Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/7R3 ingestion updates2023-08-18T11:15:09ZSiarhei Khaletski (EPAM)R3 ingestion updatesThe MR comes with:
- GONRG-1457: Stopped replacing fileSource (fix for FileSource)
- GONRG-782: add support custom acl and legal tag. update tests.
- GONRG-1402: Added Logger Section for README.md fileThe MR comes with:
- GONRG-1457: Stopped replacing fileSource (fix for FileSource)
- GONRG-782: add support custom acl and legal tag. update tests.
- GONRG-1402: Added Logger Section for README.md fileM1 - Release 0.1Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/6R3 Data Ingestion2023-08-18T11:15:11ZSiarhei Khaletski (EPAM)R3 Data IngestionThe MR comes with:
- Implemented R3 Data load;
- Data validation against schemas from Schema Service;
- Common code moved to libs folderThe MR comes with:
- Implemented R3 Data load;
- Data validation against schemas from Schema Service;
- Common code moved to libs folderM1 - Release 0.1Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/5README.md updates (GONRG-756)2023-08-18T11:15:13ZSiarhei Khaletski (EPAM)README.md updates (GONRG-756)M1 - Release 0.1Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/4GONRG-474 Unified CICD and folders structure2023-08-18T11:15:14ZOleksandr Kosse (EPAM)GONRG-474 Unified CICD and folders structureUpdate CICD pipeline for unification.
Update folders structure for unification.Update CICD pipeline for unification.
Update folders structure for unification.M1 - Release 0.1Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/3Feature/change structure and linters (GONRG-568)2023-08-18T11:15:17ZSiarhei Khaletski (EPAM)Feature/change structure and linters (GONRG-568)The MR brings the new project structure (potentially will be fine-tuned withing new commits).
Added `linters` step for deploy (`pylint` was used)
Some styles for IDE put into `.editorconfig` file (please check if your IDE/Editor supports)The MR brings the new project structure (potentially will be fine-tuned withing new commits).
Added `linters` step for deploy (`pylint` was used)
Some styles for IDE put into `.editorconfig` file (please check if your IDE/Editor supports)M1 - Release 0.1Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/1Publish manifest-based ingestion workflow (GONRG-568)2020-09-28T14:00:10ZDmitriy RudkoPublish manifest-based ingestion workflow (GONRG-568)MR Contains:
- #5: Implement custom status-tracking Airflow operator
- #6: Implement Integration tests for Ingestion DAGs
**MR highlights:**
1. MR define base project structure
2. Introduce separation between DAGs and reusable [Airflow...MR Contains:
- #5: Implement custom status-tracking Airflow operator
- #6: Implement Integration tests for Ingestion DAGs
**MR highlights:**
1. MR define base project structure
2. Introduce separation between DAGs and reusable [Airflow components](https://airflow.apache.org/docs/stable/howto/custom-operator.html):
- Operators
- Hooks
- Sensors
3. Introduce base-line for Integration tests
4. Introduce base-line for unit testsM1 - Release 0.1ethiraj krishnamanaiduJoeBrandt BealDaniel SchollAlan Brazethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/16ibm-changes2023-08-18T11:14:58ZShrikant Gargibm-changes@wladmirf @jingdongsun @ethiraj@wladmirf @jingdongsun @ethirajM3 - Release 0.5Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/14GONRG-1799: Added Azure requirements2023-08-18T11:15:00ZSiarhei Khaletski (EPAM)GONRG-1799: Added Azure requirements## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] ...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [No]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## What is the current behavior?
- Added Azure requirements (closes https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/12 conflicts)M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/13GONRG-1709: update workflow service interaction (#32)2023-08-18T11:15:01ZSiarhei Khaletski (EPAM)GONRG-1709: update workflow service interaction (#32)## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [ ] IBM
## What is the current behavior?
- Support the last contract with Workflow Service
## Have you added/updated Unit Tests and Integration Tests?
- [YES]M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/12Fix validation of entity inside the manifest (#31)2023-08-18T11:15:03ZKishore BattulaFix validation of entity inside the manifest (#31)M3 - Release 0.5https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/11Fix for pre-defined Ids and autogenerated Ids2023-08-18T11:15:05ZSiarhei Khaletski (EPAM)Fix for pre-defined Ids and autogenerated Ids## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## What is the current behavior?
- Fixed issue with pre-defined ids
- Removed Ids autogeneration
## Have you added/updated Unit Tests and Integration Tests?
- [YES]M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/10Feature/providers logic split2023-08-18T11:15:06ZSiarhei Khaletski (EPAM)Feature/providers logic split## Type of change
- [X] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X]...## Type of change
- [X] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] GCP
- [X] IBM
## What is the current behavior?
- Decoupled providers logic (please see `src/dags/providers/README.md` file)
- Fixed E2E tests
## Have you added/updated Unit Tests and Integration Tests?
- [YES]M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/9R3 updates2023-08-18T11:15:08ZSiarhei Khaletski (EPAM)R3 updates## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## What is the current behavior?
- Support of the last R3 Manifest schema
- Refreshed documentation and added sphinx docstring style.
- Added new file service ingestion flow.
## What is the new/expected behavior?
- Publish updates that makes the code vendor agnostic (it will require an additional implementation for each vendor)
## Have you added/updated Unit Tests and Integration Tests?
- [YES]M3 - Release 0.5Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/28Ingestion updates m4 (tested)2023-08-18T11:14:44ZSiarhei Khaletski (EPAM)Ingestion updates m4 (tested)## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Latest updates/fixes for M4 tag release.
Manifest ingestion was manually tested against testing Postman collectionM4 - Release 0.7Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/27GONRG-1866: add condition statement to import vendors dependencies2023-08-18T11:14:45ZSiarhei Khaletski (EPAM)GONRG-1866: add condition statement to import vendors dependencies## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Updates description?
- Added dynamic modules loading. Now there is no need to install all requirements of each vendor.M4 - Release 0.7Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/26Fix for IDs refs. Updated Docker image link2023-08-18T11:14:47ZSiarhei Khaletski (EPAM)Fix for IDs refs. Updated Docker image link## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
- Changed Docker image link
- Added validation reference IDs with versions (GONRG-1910)M4 - Release 0.7Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/24Ingestion updates2023-08-18T11:14:49ZSiarhei Khaletski (EPAM)Ingestion updates## Type of change
- [x] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x]...## Type of change
- [x] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Updates description?
This MR comes with a batch of updates:
Features:
- Validate entire manifest entity (GONRG-1783)
- Surrogate keys replacement (GONRG-1652)
- Auth logic uses Python SDK implementation (GONRG-1689)
- New operator for manifest integrity (GONRG-1700)
- New operator for schema validation (GONRG-1770)
- Logic for ensuring Datasets, WPCs and WP referencial integrity (GONRG-1653)
- Implementation of the batch uploading (GONRG-1650)
- Added FileSource validation for `Datasets` (GONRG-1651)
Structure updates:
- Removed obsolete dags (GONRG-1567)
- README.md has been updated (GONRG-1591)
- Fix `id` composing (GONRG-1700)
- Cleaned-up and renamed airflow variables (GONRG-1719)
Bugfixes:
- Handle file variable fix
- Fix download_as_bytes not supported in storage==1.13.2
- Removed `:` (colon) symbol from the end of reference ids (GONRG-1911)M4 - Release 0.7Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/22Azure credentials ut2023-08-18T11:14:51ZKishore BattulaAzure credentials ut## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Added Unit tests for azure_credentials.pyM4 - Release 0.7https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/21GONRG-1749 add manual button in pipeline for stage: deploy2023-08-18T11:14:53ZMykola Zamkovyi (EPAM)GONRG-1749 add manual button in pipeline for stage: deploy## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ]...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Updates description?
Was added a manual button in the pipeline for the stage: deployM4 - Release 0.7https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/20Ibm changes2023-08-18T11:14:55ZShrikant GargIbm changes## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] IB...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] IBM
## Updates description?
Describe your code changes in details for reviewers (links on Gitlab issues, etc.)
bug fix in ibm codeM4 - Release 0.7Anuj GuptaAnuj Gupta