Skip to content

[AZURE] File service enhancements for azure

Aman Verma requested to merge users/Vibhuti/file-enhancements-v2 into master

All Submissions:

  • [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? YES
  • [YES/NO] I have updated the documentation accordingly. YES
  • [YES/NO/NA] I have added tests to cover my changes. YES
  • [YES/NO/NA] All new and existing tests passed. YES
  • [YES/NO/NA] My code follows the code style of this project. YES
  • [YES/NO/NA] I ran lint checks locally prior to submission. NA

What is the issue or story related to the change?

Porting the file service enhancements to azure. Reference MR for GCP

!24 (merged)

High level design:

image

.Introducing concept of staging and persistence area. The end to end flow would be following

  • User will first call getLocation APIs to get upload URL
  • User will upload file using the upload URL
  • user will post the metadata using metadata API. Metadata would be stored in storage service by file service. The file would be copied from staging to persistence area. Then it'll be deleted from staging area.
  • User will call the delivery api of file service. File service would call storage service, get the file path and then generate SAS URI for the file and return this SAS URI.

Issue:

Change details:

  • Moving from one storage container model to two storage container model
  • Added implementation of following interfaces: IAuthenticationService, ICloudStorageOperation, IStorageService, IStorageUtilService
  • Added UTs for the new classes
  • Updated existing UTs with correct container names
  • Updated existing ITs with new container names
  • Updated the WIP ITs in file-test-core-bdd by removing GCP specific logic.
  • Update IT for no auth scenario same as we did for schema service to handle the response code mismatch. Refer schema-service!39 (merged)
  • Added logic to generate bearer token for azure.
  • Updated the helm charts in accordance with changes in application.properties

Test coverage:

  • Unit Test coverage is 90+ % for files touched in the MR
  • Ran both new as well as old ITs: All GREEN.

Does this introduce a breaking change?

  • [YES/NO]

The MR updates the names of Storage containers.

Pending items

Update the pipeline to run ITs inside file-test-core-bdd intead of file-test-azure whenever it's ready

Reviewer request

  • Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
  • Block the MR if you feel there is less testing or no details in the MR
  • Please cover the following aspects in the MR -- Coding design: <Reviewer1> -- Backward Compatibility: <Reviewer2> -- Feature Logic: <Logic design> -- <Any other context mention here> OR -- <Component 1>: <Reviewer1> -- <CosmosDB>: <Reviewer2> -- <ServiceBus> <Reviewer3> -- <Mention any other component and owner>

Other information

cc: @osdu/platform/roles/azure-contributors , @Vibhuti.Sharma , @devesh

Edited by Aman Verma

Merge request reports