osdu-airflow-lib merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests2022-10-09T15:57:28Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/27GONRG-3815: Make osdu_api.ini optional2022-10-09T15:57:28ZYan Sushchynski (EPAM)GONRG-3815: Make osdu_api.ini optionalThe goal of this MR is to make using `osdu_api.ini` file for `osdu_api` optional.
Now, we are able to use `Airflow variables` for initializing different `osdu_api` clients.
Also, this MR depends on https://community.opengroup.org/osdu/p...The goal of this MR is to make using `osdu_api.ini` file for `osdu_api` optional.
Now, we are able to use `Airflow variables` for initializing different `osdu_api` clients.
Also, this MR depends on https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/25M14 - Release 0.17Yan Sushchynski (EPAM)Yan Sushchynski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/30Add rc version2022-08-12T15:12:38ZYan Sushchynski (EPAM)Add rc versionM14 - Release 0.17https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/32FIX: PreloadFilePath spelling in Manifest-by-reference2022-08-31T12:05:15ZYan Sushchynski (EPAM)FIX: PreloadFilePath spelling in Manifest-by-referenceFix `PreloadFilePath` field name in Manifest-by-reference to make osdu-airflow compatible with this MR: https://community.opengroup.org/osdu/platform/system/dataset/-/merge_requests/234Fix `PreloadFilePath` field name in Manifest-by-reference to make osdu-airflow compatible with this MR: https://community.opengroup.org/osdu/platform/system/dataset/-/merge_requests/234M14 - Release 0.17https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/34Refactor Manifest-by-reference operators to be compatible with 0.17 PythonSDK...2022-09-22T12:00:14ZYan Sushchynski (EPAM)Refactor Manifest-by-reference operators to be compatible with 0.17 PythonSDK and osdu-ingestion-libRefactor Manifest-by-reference operators to be compatible with PythonSDK and osdu-ingestion-lib in 0.17Refactor Manifest-by-reference operators to be compatible with PythonSDK and osdu-ingestion-lib in 0.17M14 - Release 0.17Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/35Set no_auth parameter2022-10-25T13:12:53ZMorris EstepaSet no_auth parameterSet the no_auth parameter when uploading files to signed URLs
Closes #2Set the no_auth parameter when uploading files to signed URLs
Closes #2M14 - Release 0.17Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/36correction in schema kind authority for manifest by reference2022-10-25T12:25:41ZDevdatta Santracorrection in schema kind authority for manifest by reference## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x]...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Updates description?
This is related to issue -https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/108
Encountered an error - "Schema is not present" from Dataset service while running the manifest by reference DAG
After analysis we observed there are few codes where the 'Kind' of the schema is constructed with data-partition-id (which is 'opendes' in our case). But the schema which is present in our DB, there the 'Kind' actually starts with 'authority' (which is 'osdu' in our case). Hence the error.
So code changes done so that this value is fetched from a new DAG environment variable named "kind_authority" instead of 'data-partition-id'.M14 - Release 0.17Valentin GauthierDevdatta SantraValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/37Merge branch 'Authority_Fix_for_Manifest_By_Ref' into 'master'2022-10-26T16:53:03ZShrikant GargMerge branch 'Authority_Fix_for_Manifest_By_Ref' into 'master'correction in schema kind authority for manifest by reference
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!36
(cherry picked from commit d53ceae0a843793a001aacb77d3ca7cb8fedd349)
11e79a8a correction in schema k...correction in schema kind authority for manifest by reference
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!36
(cherry picked from commit d53ceae0a843793a001aacb77d3ca7cb8fedd349)
11e79a8a correction in schema kind authority for manifest by reference
90f30797 correction in schema kind authority
083529a3 correction in schema kind authority (review comments incorporated)
@chad Please approve this cherry-pick MR. Required for manifest by reference.
@dividoM14 - Release 0.17David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/38Merge branch '2-file-upload-is-incompatible-with-signed-urls' into 'master'2022-10-26T06:55:40ZShrikant GargMerge branch '2-file-upload-is-incompatible-with-signed-urls' into 'master'Set no_auth parameter
Closes #2
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!35
(cherry picked from commit f2d05489612700ff2daf9d735b6e30dda6340958)
65bc3ef2 Set no_auth parameter
9edd66f6 properly set data_pa...Set no_auth parameter
Closes #2
See merge request osdu/platform/data-flow/ingestion/osdu-airflow-lib!35
(cherry picked from commit f2d05489612700ff2daf9d735b6e30dda6340958)
65bc3ef2 Set no_auth parameter
9edd66f6 properly set data_partition_id
3d70fc4b add debugging messages
0160dd13 Add more debugging code.
ca4a0fef add more debugging
323b5159 fix indentation
cbee552c fix json path to signed url when using old API
c623e031 fix issue with no_auth
93161fea fix bug
9c1fb311 fix bug
ff1efc57 fix bug
158a8300 fix bug
ba31af0f fix bug
38054f26 Merge branch 'master' into 2-file-upload-is-incompatible-with-signed-urls
3c43233b remove debugging code
@chad @divido Require this cherry pick MR for manifest by referenceM14 - Release 0.17David Diederichd.diederich@opengroup.orgChad LeongDavid Diederichd.diederich@opengroup.org