Skip to content

Refactor the augmenter to search and fetch indexed data from the ElasticSearch service directly

Zhibin Mai requested to merge augmenter_refactoring_search into master

Augmenter indexing searches and fetches related data/records from the Elasticsearch Indices instead of OSDU Storage in order to complete the augmenter indexing process. Since augmenter indexing was introduced in M18, it calls the OSDU Search service instead of Elasticsearch directly to search and fetch the data. One of the reasons on choosing this strategy was to avoid overwhelming code changes by reusing the OSDU Search with easy-to-use query syntax.

However, calling OSDU Search instead of Elasticsearch does introduce unnecessary overhead on the OSDU Search which could be shared by multi-tenants. Performance-wide, it may not be a good option too.

As Augmenter becomes stable, in this MR, we refactor the Augmenter to searches and fetches related data/records via the Elasticsearch service directly. To reduce the duplicate codes between the indexer and search services and complexity, only a small portion of the codes were copied from the search service. The construction of the queries with nested query needs to be predefined.

Edited by Zhibin Mai

Merge request reports