Make system token generation more robust
Summary: There was a problem in generating a credential to access partition service from cached secret because of rotation. The Service principal is used to access Partition service for retrieving partition-specific credentials to ECK. Java service reads environment variable once at start-up. If service principal is changed (rotated) after service startup, service is unable to access Partition service. Change of service principal requires restart of service (restart of all pods) to force re-read of service principal value. This causes unnecessary down-time (returning errors or permission failures back to client).
Proposed solution: Service must be more robust to pick up new service principal value when old value fails. The token generation should attempt to retrieve the key from source if generation returns 401 after using the cached secret.