osdu-ingestion-lib merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests2023-08-18T11:25:20Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/15feat(azure): add file handing support for azure provider2023-08-18T11:25:20ZVadzim Kulybafeat(azure): add file handing support for azure providerM10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/14Revert "Merge branch 'feature/azure-file-handing' into 'master'"2023-08-18T11:25:22Zharshit aggarwalRevert "Merge branch 'feature/azure-file-handing' into 'master'"This reverts merge request !11
The tests need to be fixed before merging the changesThis reverts merge request !11
The tests need to be fixed before merging the changesM10 - Release 0.13harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/13GONRG-3773: Improve_performance2023-08-18T11:25:23ZYan Sushchynski (EPAM)GONRG-3773: Improve_performanceImprove performance of Manifest Integrity Check and Schema Validator.
Optimizations of Schema Validator make Schema Validation roughly x10 times faster.
- M9 Schema Validation of LogCurveType: 1453 sec
- New Schema Validation of LogCur...Improve performance of Manifest Integrity Check and Schema Validator.
Optimizations of Schema Validator make Schema Validation roughly x10 times faster.
- M9 Schema Validation of LogCurveType: 1453 sec
- New Schema Validation of LogCurveType: 103 secM10 - Release 0.13Siarhei Khaletski (EPAM)Aleksandr Spivakov (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/12GONRG-4000: Fix popping Payload from dag_run.conf2021-12-15T14:32:12ZYan Sushchynski (EPAM)GONRG-4000: Fix popping Payload from dag_run.confWe faced an issue when the `dag_run.conf` dictionary was modified across the tasks on Airflow>=2.2.1.
On Airflow <=2.1.2 `dag_run.conf` can't be modified.We faced an issue when the `dag_run.conf` dictionary was modified across the tasks on Airflow>=2.2.1.
On Airflow <=2.1.2 `dag_run.conf` can't be modified.M10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/11feat(azure): add file handing support for azure provider2023-08-18T11:25:25ZVadzim Kulybafeat(azure): add file handing support for azure provider## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Updates description?
Added specific header for azure provider
Details:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/issues/3
Depend on:
https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/57M10 - Release 0.13Vadzim KulybaVadzim Kulybahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/10GONRG-3789: Open VDS Metadata2021-12-14T13:53:53ZYan Sushchynski (EPAM)GONRG-3789: Open VDS MetadataAdd a class to work with Segy->* converters.Add a class to work with Segy->* converters.M10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/9GONRG-3783 common pipeline for python libs2021-12-22T18:50:42ZYan Sushchynski (EPAM)GONRG-3783 common pipeline for python libsCommon pipelinesCommon pipelinesM10 - Release 0.13Vladislav Shishko (EPAM)Vladislav Shishko (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/8Use ingestion_workflow_client2021-12-03T13:52:29ZYan Sushchynski (EPAM)Use ingestion_workflow_clientStart using the right `ingestion_workflow_client` instead of `data_workflow_client`Start using the right `ingestion_workflow_client` instead of `data_workflow_client`M10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/6Fix datetime validation2022-03-21T15:29:31ZYan Sushchynski (EPAM)Fix datetime validationDelete strict-rfc package and validation. Use data-time types as in Indexer: https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/indexer-core/src/test/java/org/opengroup/osdu/indexer/util/parser/DateTimePar...Delete strict-rfc package and validation. Use data-time types as in Indexer: https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/indexer-core/src/test/java/org/opengroup/osdu/indexer/util/parser/DateTimeParserTest.java#L31
closes #2M10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/4GONRG-3618: Local build setup2021-11-03T11:01:35ZYan Sushchynski (EPAM)GONRG-3618: Local build setupResolves the issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/issues/1, when it was needed to specify extra environmental variables if the package was built locally.Resolves the issue: https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/issues/1, when it was needed to specify extra environmental variables if the package was built locally.M10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/2GONRG-3119: Python SDK Usage2022-01-07T09:03:32ZYan Sushchynski (EPAM)GONRG-3119: Python SDK UsageAdd Python SDK `clients` instead of `requests`.Add Python SDK `clients` instead of `requests`.M10 - Release 0.13Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)