indexer worker API is broken if Enterprise license expires
We have observed failures on indexing requests when Elasticsearch server Enterprise license expires. In such cases, Enterprise license is automatically downgraded to trial license and with that, we see multiple restrictions enforced on the Elastic API. Here is summary from logs:
"LICENSE [EXPIRED] ON [THURSDAY, SEPTEMBER 19, 2024].\n
# IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT. OTHERWISE, PLEASE REACH OUT TO\n# YOUR SUPPORT CONTACT.\n# \n
# COMMERCIAL PLUGINS OPERATING WITH REDUCED FUNCTIONALITY\n
# - security\n
# - Cluster health, cluster stats and indices stats operations are blocked\n
# - All data operations (read and write) continue to work\n
# - watcher\n
# - PUT / GET watch APIs are disabled, DELETE watch API continues to work\n
# - Watches execute and write to the history\n
# - The actions of the watches don't execute\n
# - monitoring\n
# - The agent will stop collecting cluster and indices metrics\n
# - graph\n
# - Graph explore APIs are disabled\n
# - ml\n
# - Machine learning APIs are disabled\n
# - logstash\n
# - Logstash will continue to poll centrally-managed pipelines\n
# - beats\n
# - Beats will continue to poll centrally-managed configuration\n
# - deprecation\n
# - Deprecation APIs are disabled\n
# - upgrade\n
# - Upgrade API is disabled\n
# - sql\n
# - SQL support is disabled\n
# - rollup\n
# - Creating and Starting rollup jobs will no longer be allowed.\n
# - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function.\n
# - transform\n
# - Creating, starting, updating transforms will no longer be allowed.\n
# - Stopping/Deleting existing transforms continue to function.\n
# - analytics\n
# - Aggregations provided by Analytics plugin are no longer usable.\n
# - ccr\n
# - Creating new follower indices will be blocked\n
# - Configuring auto-follow patterns will be blocked\n
# - Auto-follow patterns will no longer discover new leader indices\n
# - The CCR monitoring endpoint will be blocked\n
# - Existing follower indices will continue to replicate data"
While triaging the issue, we found Elasticsearch blocking cluster health request when license is downgraded to trial.
Indexer has an implementation on indexing request to check the health (via API) of index/shards before start calling batch indexing API. All this is done by index-worker API.