ADR: Search backend (Elasticsearch) Upgrade

Status

• Proposed
• Under review
• Approved
• Retired

Background

Elasticsearch serves as the backend for indexer and search services. To communicate with the Elasticsearch server (deployed and managed independently), these services use the Elasticsearch Java high level rest API client SDK. Official OSDU Data Platform supported Elasticsearch version for server & client SDK is v7.8.1. Current version is quite old, and already beyond end of life support. A new major version (v.8.x), which was released in April 2022, is also available. Furthermore, not only will an upgrade to Elasticsearch resolve issues & offers new features and capabilities, but it will also save costs. Here are few reasons, Elasticsearch client & server should be updated:

• Log4J vulnerability discovered on December 2021 forced all CSPs to update their Elasticsearch server version. At this time, all CSPs are on different server versions e.g. Azure v7.17.x, IBM v7.11.x etc. Even though Elasticsearch promises on not introducing any breaking change on a major version, we have found issues in past. Ideally all CSPs should be on same client & server versions to avoid any potential issues.

  Community effort on Reference Implementation gives us a good opportunity to upgrade and align Elasticsearch client and server version.

• Elasticsearch v7.8.1 has reached end of life some time back. Officially supported version for Elasticsearch v7 is v7.17.x or higher. If an issue found with client SDK or server, than fix is usually avaialable in most recent version.

• Elasticsearch has launched many new versions past v7.8.1 with several improvements & new features, some notable ones in v8.x are mentioned below:

  • Elasticsearch v8.3.x has removed 1k shard count per node limitations. OSDU DD Definition Team has introduced several new schemas over the course of few milestone releases. On Elasticsearch, each schema index generates two shards. Currently, a single node in an elasticsearch instance can only hold up to 1K shards. A small or medium-sized Elasticsearch cluster can quickly run out of shard capacity with so many new schemas.
  • Reduced resource requirements via memory heap reductions. This can result in lowered customers’ total cost of ownership. Added support for the ARM architecture, it offers 20% better performance while being 10% cheaper than x86-64. Introduced novel ways to use less storage by decoupling compute from storage with a new frozen tier and searchable snapshots.
  • Improved indexing latency of several data types including geo-points, geo-shapes etc. Enhanced error messages on invalid geo-shape indexing. It can now provide more meaningful messages capturing issues with shape, rather a generic messages in current version. Several new geo queries (e.g. geo-grid query etc.), aggregations (e.g. cartesian-centroid, geo-hex aggregation over points and shapes etc.) are also introduced.
  • Introduced a new health API designed to report the health of the cluster. The new API offers a detailed report that can include a precise diagnosis and a solution, as well as a high level overview of the cluster health. The operational teams can benefit greatly from this API.
  • Released a full suite of native vector search via kNN search. It adds support for natural language processing (NLP) models directly into Elasticsearch. Users can now perform named entity recognition, sentiment analysis, text classification, and more directly in Elasticsearch — without requiring additional components or coding. Elasticsearch v8.x also includes native support for approximate nearest neighbor (ANN) search — making it possible to compare vector-based queries with a vector-based document corpus with speed and at scale.

Proposal

Any Elasticsearch upgrade will require coordination with community and CSPs. This can be very time consuming. Instead of just upgrading Elasticsearch to latest v7.17.x, we should upgrade to v8.10.0 (or the highest released v8.x) to minimize the disruption and repeat this step very soon. Since the last major version of Elasticsearch (v8) was released 18 months ago, once v9 is released, the entire v7 (v7.17.x) family will be deprecated, as stated in the support documentation.

We should breakdown upgrade into two parts:

Latest v7.17.x Upgrade

1. Take back up (snapshot) of the data. We cannot roll back to an earlier version unless we have snapshot.

2. Upgrade Elasticsearch server to latest v7.17.22 (or highest available v7.17.x).

3. Replace Indexer & Search services Java high level rest API client SDK with new Java API client SDK. Elasticsearch has changed Java high level rest API client SDK's license in v7.10.2 from Apache 2.0 to SSPL. New license is not preffered license for OSDU Data Platform as explained in the issue.

   Java API client SDK with Apache 2.0 license is available v7.15.0 onwards (including v.8.x). Along similar timeline Java high level rest API has been deprecated in favor of Java API client SDK.

Latest v8.15.x Upgrade

1. Complete client and server upgrade to v7.17.22 or higher.

2. Use the Kibana Upgrade Assistant to prepare for upgrade from v7.17 to v8.15.0. The Upgrade Assistant identifies deprecated settings and guides users through resolving issues.

   

3. Review the deprecation logs from the Upgrade Assistant to determine if your applications are using features that are not supported or behave differently in 8.15.0.

   All critical Elasticsearch and Kibana configuration issues must be resolved before upgrading to Elastic 8.x. Ignoring warnings might result in differences in behavior after you upgrade

   

   These issues can be different for providers depending on the Elasticsearch or Kibana configuration they are using.

4. Take a current snapshot before server upgrade is started.

5. Upgrade Elasticsearch server to latest v8.15.2 (or highest available v8.x). Elasticsearch upgrade recommendations from v7.x to v8.x can be found here.

   1. If you are using Elastic cloud, upgrade instructions can be found here.
   2. If you are using self hosted/on-prem Elasticsearch, upgrade instructions can be found here.

6. Test the upgrade in an isolated environment before upgrading your production cluster.

7. Set the REST API compatibility for all requests received by Elasticsearch by setting the environment variable ELASTIC_CLIENT_APIVERSIONING to true.

8. Review the deprecation logs for entries with the category compatible_api. Review the workflow associated with the requests that relied on compatibility mode.

9. Upgrade Indexer and Search service Java API client SDK to v8.15.2 (or equal to Elasticsearch server version in above step) and resolve compatibility issues manually where needed.