Skip to content

Adding Circuit Breaker to the service calls.

SHEFFALI JAIN requested to merge addingCB into Azure/OSDU-Helm-Charts-Azure-M8-Master

Circuit Breaker provides a mechanism to fail fast in case there is any failure in one of our micro-services to prevent cascading failure.

Implemented using Istio.

ARCHITECTURE:

As circuit breaking applies to “real” mesh destinations in a load balancing pool, we configure circuit breaker thresholds in destination rules, with the settings applying to each individual host in the service. Hosts that continually return 5xx errors(or defined error) for API calls are ejected from the pool for a pre-defined period of time.

CODE SNIPPET: image This configures upstream hosts to be scanned every 20 secs so that any host that fails 1 consecutive times with a 5xx (502, 503, etc) error code will be ejected for 2 minutes.

Refer here for more errors that can be customized for circuit breaking : https://istio.io/latest/docs/reference/config/networking/destination-rule/

CB conditions can be applied at :

  • consecutiveLocalOriginFailures
  • consecutiveGatewayErrors
  • consecutive5xxErrors

image Test Scenarios

  1. Ensure circuit breaks only for that service which is down.
  2. Ensure Retries are handled well with circuit breakers.
Edited by SHEFFALI JAIN

Merge request reports