Fix Thread Exhaustion and ES Connection Errors

Fix for Issue: #240 (closed) and #205

Implemented Fixes

1. Replaced Elasticsearch client cache with request-scoped client.

Description

Current Implementation

The existing Elasticsearch client cache was used to avoid the re-instantiating the client for each request to Elasticsearch, which could happen several times during an indexing operation. Instantiating new clients is an expensive operation, which is why these clients are cached. Each client is configured for a specific Elasticsearch database, which could be different for each partition, which is why the clients are cached per partition.

Fix Rationale

Issue #240 (closed) can cause the Indexer pod to crash due to out-of-memory or will fail to create new rest clients because the OS has reached it's thread limit. This is caused by the caching mechanism. The original cache expired leaving orphaned clients not cleaned up in memory. Using a cache without expiry appeared to address the issue but the threads would still continue to accumulate over time leading to the same issue above.

The cause of this was the low-level rest client transport. The Elasticsearch client would continue to create new connections but would not close them after a request was complete. The challenge is how to manage the lifecycle of the connections using a shared client:

• Creating and closing each request is expensive considering multiple requests to Elasticsearch are made per request.
• Closing the connection while the shared client is in use by another request thread causes other requests to fail.
• Using a shared connection is not possible since it blocks requests in other threads causing the requests to fail.
• Batching Elasticsearch requests would add latency to each request and add significant complexity.
• Using a client pool instead of a cache would allow re-use of clients but add complexity in the implementation. The health of the client would need to be checked before returning to the pool. Unused clients would need to be cleaned up on a schedule. A connection pool would be required per partition (or per ES database) since each client could be configured differently by partition.

Considering that this is a hot-fix for an M25 indexing issue, I instead chose a simple request-scoped approach where a new ES client is created during the initial request, re-used for each request to ES, and properly closed once the request is finished. This is the simplest approach which avoids caching clients by partition and other complex lifecycle management features that would be required by a connection pool.

Testing

I monitored an indexer pod after the proposed changes during indexing 130k+ records (TNO/Volve/Reference dataset). I did not observe any OOM and very few connection cancellation exceptions mentioned in issue: #205. 100% of records were indexed successfully; indexing speed appears to be 20-30% faster. Maximum thread count observed was 78 across all running pods.

Detailed metrics from longest running pod:

Executing top (resource usage):

  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    7     1 appuser  S    2890m  37%   1   1% java -Xms1000M -Xmx1000M --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED -jar /app.jar
  231     0 appuser  S     2984   0%   0   0% bash
    1     0 appuser  S     1712   0%   0   0% /bin/sh -c . /entrypoint.sh
  350   231 appuser  R     1700   0%   3   0% top

Executing ps -lfT | wc -l (thread count):

78