Discrepancy in Storage API for create/update record operation

For Storage create/update record API, if a record ID ends in a dot (.) the data block for the record is not properly uploaded to the Microsoft storage account. In cases, where the records for create/update multiple record operation have the similar IDs only differentiated by a dot at the end (ex. M/M.), the data block will be the same for both records. The issue is that Microsoft storage accounts do not support directory names ending with a dot (.), a forward slash (/), or a backslash (\) and path segments ending with a dot (https://learn.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata). When uploading the block blob with the record data to the storage container, the BlobStore.class tries to use a path with the record ID as a folder, such as

<kind>/<partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M./1704916580557751

but <partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M. is not a valid directory name so the dot at the end is ignored, and block blob is uploaded to <partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M instead. This was also manually confirmed by trying to upload a blob to a folder with a name ending in dot.

It is a corner case but this issue has impacted RDD values for M and M. on all partitions

  • For example on "prod-weu-des-prod-testing-eu", the records prod-weu-des-prod-testing-eu:reference-data--ExternalUnitOfMeasure:LIS-LAS::M. and prod-weu-des-prod-testing-eu:reference-data--ExternalUnitOfMeasure:LIS-LAS::M has same "M." values in ID, Code and Symbol field. These were created using RDD script/pipeline.
  • Impact is on all partitions for <partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M. and <partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M values 

Proposed solution is to reject record IDs that end in these unsupported characters (i.e. return 400 bad request when such record IDs are used).

Edited Jan 11, 2024 by Neha Khandelwal
Assignee Loading
Time tracking Loading