feat: compute size service implementation
Type of change
-
Bug Fix -
Feature -
Pipeline -
Test -
Documentation
Does this introduce a change in the core logic?
-
No -
Yes
Does this introduce a change in the cloud provider implementation, if so which cloud?
-
AWS -
Anthos -
Azure -
GCP -
IBM
Does this follow conventional commits spec?
-
No -
Yes
Have you set the target Milestone?
-
No -
Yes
Have you set the no-detached-pipeline label?
-
No -
Yes
Updates description?
This MR contains an implementation of ComputeSize background service (Azure), that is responsible for syncing the size property between Azure Storage and datasets metadata. The compute size operation is triggered on dataset being closed (using PATCH API endpoint 'close'). Then the message is sent to a Azure Storage queue and picked up from a queue by the ComputeSize service. The ComputeSize service fetches blobs for the dataset, sums up their sizes and compares the size with the one sent in the message queue. If they are not the same then it locks the dataset and performs a partial update on the dataset metadata record.
E2E tests will be raised in the next MR (after updating infrastructure in helm-charts-azure repo).