Discrepancy in Storage API for create/update record operation
For Storage create/update record API, if a record ID ends in a dot (.) the data block for the record is not properly uploaded to the Microsoft storage account. In cases, where the records for create/update multiple record operation have the similar IDs only differentiated by a dot at the end (ex. M/M.), the data block will be the same for both records. The issue is that Microsoft storage accounts do not support directory names ending with a dot (.), a forward slash (/), or a backslash (\) and path segments ending with a dot (https://learn.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata). When uploading the block blob with the record data to the storage container, the BlobStore.class tries to use a path with the record ID as a folder, such as
<kind>/<partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M./1704916580557751
but <partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M. is not a valid directory name so the dot at the end is ignored, and block blob is uploaded to <partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M instead. This was also manually confirmed by trying to upload a blob to a folder with a name ending in dot.
It is a corner case but this issue has impacted RDD values for M and M. on all partitions
- For example on "prod-weu-des-prod-testing-eu", the records prod-weu-des-prod-testing-eu:reference-data--ExternalUnitOfMeasure:LIS-LAS::M. and prod-weu-des-prod-testing-eu:reference-data--ExternalUnitOfMeasure:LIS-LAS::M has same "M." values in ID, Code and Symbol field. These were created using RDD script/pipeline.
- Impact is on all partitions for <partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M. and <partition>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M values
Proposed solution is to reject record IDs that end in these unsupported characters (i.e. return 400 bad request when such record IDs are used).