put_file function in dataset_dms_client and Azure storage client implementation.
Type of change
-
Bug Fix -
Feature
Does this introduce a change in the core logic?
- [Yes]
Does this introduce a change in the cloud provider implementation, if so which cloud?
-
AWS -
Azure -
GCP -
IBM
Updates description?
With reference to suggested changes in the previous MR,
osdu/platform/data-flow/ingestion/osdu-airflow-lib!48 (comment 198135)
I've made below changes.
-- Added new function call as put_file for Uploading data on dataset_service.
put_file will also check the CLOUD_PROVIDER and perform provider specific operation if is there any then the make_request call.
Location: dataset_dms_client.py
-- Implemented storage client for Azure.
(Implemented Functions: _does_file_exist_, _download_to_file_, _download_file_as_bytes_, _upload_file_
)
Location: azure_blob_storage_client.py
Note: Call to the put_file will be from osdu_airflow_lib once above independent changes are merged osdu_airflow_lib changes can be merged
Edited by Jayesh Bagul