Skip to content

put_file function in dataset_dms_client and Azure storage client implementation.

Jayesh Bagul requested to merge az/jb_add_storage_client_put_file into master

Type of change

  • Bug Fix
  • Feature

Does this introduce a change in the core logic?

  • [Yes]

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Azure
  • GCP
  • IBM

Updates description?

With reference to suggested changes in the previous MR,
osdu/platform/data-flow/ingestion/osdu-airflow-lib!48 (comment 198135)
I've made below changes.

-- Added new function call as put_file for Uploading data on dataset_service.
put_file will also check the CLOUD_PROVIDER and perform provider specific operation if is there any then the make_request call.
Location: dataset_dms_client.py

-- Implemented storage client for Azure.
(Implemented Functions: _does_file_exist_, _download_to_file_, _download_file_as_bytes_, _upload_file_)
Location: azure_blob_storage_client.py

Note: Call to the put_file will be from osdu_airflow_lib once above independent changes are merged osdu_airflow_lib changes can be merged

Edited by Jayesh Bagul

Merge request reports