Refactor the augmenter to search and fetch indexed data from the ElasticSearch service directly
Augmenter indexing searches and fetches related data/records from the Elasticsearch Indices instead of OSDU Storage in order to complete the augmenter indexing process. Since augmenter indexing was introduced in M18, it calls the OSDU Search service instead of Elasticsearch directly to search and fetch the data. One of the reasons on choosing this strategy was to avoid overwhelming code changes by reusing the OSDU Search with easy-to-use query syntax.
However, calling OSDU Search instead of Elasticsearch does introduce unnecessary overhead on the OSDU Search which could be shared by multi-tenants. Performance-wide, it may not be a good option too.
As Augmenter becomes stable, in this MR, we refactor the Augmenter to searches and fetches related data/records via the Elasticsearch service directly. To reduce the duplicate codes between the indexer and search services and complexity, only a small portion of the codes were copied from the search service. The construction of the queries with nested query needs to be predefined.
Merge request reports
Activity
changed milestone to %M25 - Release 0.28
added Common Code MREnhancement labels
assigned to @zmai
requested review from @Stanislav_Riabokon, @Rustam_Lotsmanenko, @ydzeng, and @nthakur
added 1 commit
- 98581211 - Add unit test to detect any change on the original object caused by method...
added 1 commit
- 607654e9 - Fix the potential issue on search with search_after and pit
@ydzeng Thanks!
requested review from @mzhu9
@kimjiman @jordanchiu Thanks!
requested review from @jordanchiu and @kimjiman
@Stanislav_Riabokon Thanks!
mentioned in commit dfb961e7
