feat: use optimised CosmosDB queries to retrieve subfolders

Type of change

• Bug Fix
• Feature
• Pipeline
• Test
• Documentation

Does this introduce a change in the core logic?

• No
• Yes

Does this introduce a change in the cloud provider implementation, if so which cloud?

• AWS
• Anthos
• Azure
• GCP
• IBM

Does this follow conventional commits spec?

• No
• Yes

Have you set the target Milestone?

• No
• Yes

Have you set the no-detached-pipeline label?

• No
• Yes

Updates description?

This PR introduces an optimised queries to fetch datasets and subfolders for Azure provider. Changes:

• changing the query to fetch subfolders into a query that fetches distinct paths
• add a feature flag for enabling the optimised subfolders query ENABLE_OPTIMISED_QUERY (true by default)
• add unit tests for resolving the query for listFolders