Key Vault quota exceed issue faced in case of high volume ingestion
We are facing an issue of KeyVault quota exceed when high volume of ingestion workflows triggered,
CSV parser publish events to status topic using methods from azure core lib’s publisher façade.
There are two strategies to connect to service bus One uses Default credentials with service bus namespace and another via service bus connection string Both service bus namespace and service bus connection string are secrets in KeyVault and fetched via Partition service
For rest of the services it is getting cached so should not be an issue.
But in case of Airflow where for each CSV workflow job run a new POD is instantiated by Airflow, Key Vault is accessed for each job. Azure core lib caching here is irrelevant.
We observed that in case of high influx of jobs Key Vault is accessed very frequently in a short period of time results into many KeyVault access quota exceed issue.
Our suggestion is we can keep service bus namespace as a non secret value and use Default credentials with service bus namespace to create topic client.