Filtering support for bulk deletion

Type of change

• Bug Fix
• Feature
• Pipeline
• Test
• Documentation

Does this introduce a change in the core logic?

• No
• Yes

Does this introduce a change in the cloud provider implementation, if so which cloud?

• AWS
• Anthos
• Azure
• GCP
• IBM

Does this follow conventional commits spec?

• No
• Yes

Have you set the target Milestone?

• No
• Yes

Have you set the no-detached-pipeline label?

• No
• Yes

Updates description?

Update this part with a description of your changes

• Refactors the listDatasets() function implementation to first generate a query and then list the datasets.
• Adds a new listDatasetsQuery() that returns a query SQL.
• Encapsulates the parameters used in listDatasets() in a ListDatasetsParams object; adds a new recursive parameter which allows listing datasets recursively also in subfolders.
• Adds a STARTSWITH filter to the Cosmos DB implementation to support listing datasets recursively.
• Adds a query retrieving datasets to the Redis queue message that triggers a bulk-delete job. Deletes the unused path from the message; the remaining fields (tenant, subproject) have not been deleted from the message as they are required in the deletion runner to obtain locks for the datasets.
• Updates the deletion runner to use the ready query instead of a hardcoded one to fetch datasets for deletion.