Poor performance for index augmenter

Though we made several enhancements related to index augmenter directly or indirectly, such as creating separating re-index topic, splitting the big message with 1000 records to small message with 50 records to support parallel indexing, and etc. We still found that the index performance with augmenter enabled is much worse than the index performance with augmenter disabled. For example, for WellLog with multiple extension configurations, the performance with augmenter enabled is about 15 times slower than the performance with augmenter disabled.

With augmenter enabled,

1. Index one record individually, each record (for given property configurations) requires 8 queries to get all information in order to populate the extended properties. In this test test, cache does not take effect at all.
2. Index a kind with 291 WellLog records, each record requires 6.8 queries on average. In this test case, the cache should play important role. However, we found the cache mechanism basically does not take much effect.

As I ran the tests from local, the latency of search is about 1.5 times longer than the latency of search in cloud env. I estimated that the performance with augmenter enabled is still about 10 times slower if we don't make any enhancement.