fix: split batchclient delete request into multiple
Type of change
-
Bug Fix -
Feature -
Pipeline -
Test -
Documentation
Does this introduce a change in the core logic?
-
No -
Yes
Does this introduce a change in the cloud provider implementation, if so which cloud?
-
AWS -
Anthos -
Azure -
GCP -
IBM
Does this follow conventional commits spec?
-
No -
Yes
Have you set the target Milestone?
-
No -
Yes
Have you set the no-detached-pipeline label?
-
No -
Yes
Updates description?
MR splits the batchclient's delete request into multiple. The batchClient from Azure SDK has a limit of 256 changes per request, if a dataset has more than 256 blobs it will have an error. The cosmosDB metadata is deleted before contacting storage to delete the blobs, so now the blobs are orphaned.
For large datasets should use the builk-delete operation endpoint