Skip to content

Filtering support for bulk deletion

Type of change

  • Bug Fix
  • Feature
  • Pipeline
  • Test
  • Documentation

Does this introduce a change in the core logic?

  • No
  • Yes

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Anthos
  • Azure
  • GCP
  • IBM

Does this follow conventional commits spec?

  • No
  • Yes

Have you set the target Milestone?

  • No
  • Yes

Have you set the no-detached-pipeline label?

  • No
  • Yes

Updates description?

Update this part with a description of your changes

  • Refactors the listDatasets() function implementation to first generate a query and then list the datasets.
  • Adds a new listDatasetsQuery() that returns a query SQL.
  • Encapsulates the parameters used in listDatasets() in a ListDatasetsParams object; adds a new recursive parameter which allows listing datasets recursively also in subfolders.
  • Adds a STARTSWITH filter to the Cosmos DB implementation to support listing datasets recursively.
  • Adds a query retrieving datasets to the Redis queue message that triggers a bulk-delete job. Deletes the unused path from the message; the remaining fields (tenant, subproject) have not been deleted from the message as they are required in the deletion runner to obtain locks for the datasets.
  • Updates the deletion runner to use the ready query instead of a hardcoded one to fetch datasets for deletion.
Edited by Maggie Salak

Merge request reports