Skip to content

Add circuitbreaker to all inter service communication

Ronak Sakhuja requested to merge rosakhuj/circuitbreakerdecorate into master

All Submissions:


  • [YES] I have added an explanation of what changes in this merge do and why we should include it?
  • [NO] I have updated the documentation accordingly.
  • [YES] I have added tests to cover my changes.
  • [YES] All new and existing tests passed.
  • [YES] My code follows the code style of this project.
  • [YES] I ran lint checks locally prior to submission.

What is the issue or story related to the change?


We are adding a circuitbreaker to all Interservice communication. All Services use HTTPClient's send method to make requests. In this MR, we are decorating the send method with a circuibreaker to avoid the problem of cascading fallback. Be default, we are using default configurations of circuibreaker. Values can be found here(https://resilience4j.readme.io/docs/circuitbreaker#create-and-configure-a-circuitbreaker)

High level design: image CircuitBreaker is at the service level. At pod level communication, the URL for let's say entitlements service is defined as : http://entitlements/api/entitlements/v2 . We will fetch the host and make it as the key for circuitbreaker.

Issue:

Change details: We are adding a circuitbreakersingleton class to keep the circuitbreakerregistry. There is a flag which can be set to enable circuitbreaker for a service. When circuitbreaks, it will start throwing error 500 without making http call.

Test coverage & Tests:


Used Storage service and Entitlements service for testing. While Entitlements was up, sent HttpRequests from Storage to Entitlements with success response. Turned off Entitlements service and then sent requests from storage to entitlements which resulted in errors. After the required number of calls, obtained log that circuit has opened and post which the requests were failing fast.

Tested the behavior by running on kubernetes cluster locally.

Test on perf environment

I setup Storage and Entitlements on perf, and added a preauthorize on healthcheck in storage. Scenario : Storage service calling entitlements

  1. Behaviour when Entitlements is up : image
  2. Behaviour when Entitlements is down and circuit is closed : image
  3. Behaviour when Failure threshold is crossed and circuit is open : image

UPDATE : When Circuit is open, we are throwing Exception 503 : Service Unavailable

Logs generated when circuit gets open : image

Tested to ensure multiple circuitbreakers for different services can be maintained.

Does this introduce a breaking change?


  • NO

Pending items


  1. Make all configuration of circuitbreaker configurable.
  2. Segregate circuitbreakers to remove parameters from circuitbreaker name

Reviewer request


  • Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
  • Block the MR if you feel there is less testing or no details in the MR
  • Please cover the following aspects in the MR -- Coding design: <Reviewer1> -- Backward Compatibility: <Reviewer2> -- Feature Logic: <Logic design> -- <Any other context mention here> OR -- <Component 1>: <Reviewer1> -- <CosmosDB>: <Reviewer2> -- <ServiceBus> <Reviewer3> -- <Mention any other component and owner>

Other information


Edited by Ronak Sakhuja

Merge request reports