Indexer failures with 5XX error codes should be searchable
Some of the records indexing fails due to unknown (5xx) error. As of today, these records are then not searchable, not identified as errors during indexing. Records failed with 5XX errors should be searchable with default indexing fields.
Context
Currently, when the schema or storage service throws a 500 error, indexer stops right away. After a few retries, indexer ignores this record. Thus, the records that failed with index status 500 are not searchable.
Proposed solution
We can use the record changed/updated event to also contain the acl and legal tag section of the record. This will enable indexer to index the failed record (assuming no schema is exists) with the the right metadata, thereby making the record searchable. Indexer will use the id, kind, acl and tags properties to index this record and the trace will be available to view as a result of the query "index.status = 500".