503 error "upstream connect error"

Sometimes entitlements v2 GET groups endpoint (Azure cloud) returns 503 error with the body "upstream connect error or disconnect/reset before headers, reset reason: connection failure".

According to https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace Pod termination includes 3 steps:

1. Pod is set to the “Terminating” State and removed from the endpoints list of all Services
2. PreStop Hook is executed: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-details
3. SIGTERM signal is sent to the pod

If PreStop is empty then the 1st and 2nd steps are executed instantaneously.

Then the entitlements application gets SIGTERM and stops immediately.

There is a “server.shutdown” property for Spring Boot but its default value is “immediate” (not “graceful”).

Therefore if there are active connections they will be terminated immediately after SIGTERM.