KV throttling issue - improve caching
All Submissions:
- [YES] I have added an explanation of what changes in this merge do and why we should include it?
- [NA] I have updated the documentation accordingly.
- [YES] I have added tests to cover my changes.
- [YES] All new and existing tests passed.
- [YES] My code follows the code style of this project.
- [YES] I ran lint checks locally prior to submission.
What is the issue or story related to the change?
Issue: If the system is under moderate/heavy load, we are seeing a lot of issues with Storage service because of KV throttling. The error we see is:
storage.app Status code 429, "{"error":{"code":"Throttled","message":"Request was not processed because too many requests were received. Reason: VaultOperationLimitReached"}}"
Change details: To address this, we can follow the kv-throttling guideline from MSFT which involves:
- Retry policy when creating the secret client (EDIT: Removed after comments, to be considered in a separate MR)
- Improve caching when Storage calls
publishToEventGridTopic
Test coverage:
Does this introduce a breaking change?
- [NO]
Pending items
Reviewer request
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR -- Coding design: <Reviewer1> -- Backward Compatibility: <Reviewer2> -- Feature Logic: <Logic design> -- <Any other context mention here> OR -- <Component 1>: <Reviewer1> -- <CosmosDB>: <Reviewer2> -- <ServiceBus> <Reviewer3> -- <Mention any other component and owner>