Fixing Azure reindex so that it is able to interact with entitlements service of http within the k8s cluster. Related to #14 (closed)
From initial investigation, it seems like the issue was in part that the x-functions-key header was not being set properly, and so the value was coming through as null when indexer-service was checking authorization with entitlements. This is because Istio was preventing the request from even hitting the service container for entitlements. When we turn off Istio, we see some informative error logs from tomcat in the entitlements pod:
2021-03-17 13:11:46.830 INFO entitlements-azure-57764d67f8-6jthb --- [p-nio-80-exec-4] o.a.c.h.Http11Processor correlation-id= data-partition-id=: Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: The HTTP header line [x-functions-key0x0d] does not conform to RFC 7230 and has been ignored.
at org.apache.coyote.http11.Http11InputBuffer.skipLine(Http11InputBuffer.java:1041)
at org.apache.coyote.http11.Http11InputBuffer.parseHeader(Http11InputBuffer.java:893)
at org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:593)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:284)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:880)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1601)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)This is interesting for two reason: (1) it seems that Istio and Tomcat perform some header validation. (2) For whatever reason, the validation is only behaving this way over http and not https. I did a bit of reading on the Tomcat code that is throwing the error. I think the relevant part is here. So since there is this illegal character in the header, Tomcat throws a 400 on behalf of entitlements service for the request. It seems Istio must do something similar since the request does not even reach the service container when Istio is on.
Interestingly, both Istio and Tomcat only seem to be doing this for http and not https. We were able to successfully make the call (even with the null header) before when we referred to entitlements service by its DNS name rather than the internal access point within the cluster (http://entitlements-azure/entitlements/v1).
This MR resolves this issue by directly hardcoding the value for the x-functions-key header. We were already hardcoding the value to be abcd in application.properties. However, it was not being set properly because the code block the defines what the header value should be set to:
@Value("${indexer.queue.key}")
private String queueKey;was running after the variable was used, so it was coming through as null.