Skip to content

feat: compute size service implementation

Izabela Kulakowska requested to merge msft/feat/compute-size into master

Type of change

  • Bug Fix
  • Feature
  • Pipeline
  • Test
  • Documentation

Does this introduce a change in the core logic?

  • No
  • Yes

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Anthos
  • Azure
  • GCP
  • IBM

Does this follow conventional commits spec?

  • No
  • Yes

Have you set the target Milestone?

  • No
  • Yes

Have you set the no-detached-pipeline label?

  • No
  • Yes

Updates description?

This MR contains an implementation of ComputeSize background service (Azure), that is responsible for syncing the size property between Azure Storage and datasets metadata. The compute size operation is triggered on dataset being closed (using PATCH API endpoint 'close'). Then the message is sent to a Azure Storage queue and picked up from a queue by the ComputeSize service. The ComputeSize service fetches blobs for the dataset, sums up their sizes and compares the size with the one sent in the message queue. If they are not the same then it locks the dataset and performs a partial update on the dataset metadata record.

E2E tests will be raised in the next MR (after updating infrastructure in helm-charts-azure repo).

Edited by Izabela Kulakowska

Merge request reports