Skip to content

KV throttling issue - improve caching

Alok Joshi requested to merge kv_throttling into master

All Submissions:

  • [YES] I have added an explanation of what changes in this merge do and why we should include it?
  • [NA] I have updated the documentation accordingly.
  • [YES] I have added tests to cover my changes.
  • [YES] All new and existing tests passed.
  • [YES] My code follows the code style of this project.
  • [YES] I ran lint checks locally prior to submission.

What is the issue or story related to the change?

Issue: If the system is under moderate/heavy load, we are seeing a lot of issues with Storage service because of KV throttling. The error we see is: storage.app Status code 429, "{"error":{"code":"Throttled","message":"Request was not processed because too many requests were received. Reason: VaultOperationLimitReached"}}"

Change details: To address this, we can follow the kv-throttling guideline from MSFT which involves:

  • Retry policy when creating the secret client (EDIT: Removed after comments, to be considered in a separate MR)
  • Improve caching when Storage calls publishToEventGridTopic

Test coverage:

Does this introduce a breaking change?

  • [NO]

Pending items

Reviewer request

  • Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
  • Block the MR if you feel there is less testing or no details in the MR
  • Please cover the following aspects in the MR -- Coding design: <Reviewer1> -- Backward Compatibility: <Reviewer2> -- Feature Logic: <Logic design> -- <Any other context mention here> OR -- <Component 1>: <Reviewer1> -- <CosmosDB>: <Reviewer2> -- <ServiceBus> <Reviewer3> -- <Mention any other component and owner>

Other information

Edited by Alok Joshi

Merge request reports