Use specific topic instead of the storage record change topic to send the re-index events
In current implementation of Azure indexer, re-index events share the same topic of the storage record change events. It creates several kinds of problems:
- Create unnecessary load on the storage service as many other services monitor the storage change events and react, e.g. data synch with external datastores
- It could affect the index/re-index performance if storage service is busy
- Create unnecessary duplicate copies of the data, e.g. multiple copies/versions of wks records with extract same content could be created.
- Events generated from re-index or index-extension could block storage record change events which could have impact on SLO requirements in terms of index update latency.
We should use specific topic for re-index to send and receive the re-index events.
Edited by Zhibin Mai