Storage API /query/kinds is broken and breaks reindex functionality
**Takeaway**
The /query/kinds API has been broken in OSDU Storage for quite a while, and fixing it was not a priority as Schema Service endpoints were thought to be the successor solution. This is not the case, and /query/kinds needs to work as designed.
Summary
The context here is the issue to change the Indexer to use Schema Service schemas instead of the original Storage Schemas (indexer-service#7 (closed)). This has been done successfully; however, the original plan to retire the Storage Schema endpoints + /query/kinds entirely exposed a hole in functionality that needs to be addressed. Essentially, it was thought that fixing /query/kinds would not be needed with the Schema Service, but the use cases where Storage is the source of truth for in use kinds were not caught.
Key Use Case -- reindexing all kinds
Reindexing all kinds in an Elasticsearch cluster (Reindex All) is an infrequent but vital operation. Cases where it is required include:
disaster recovery after Storage Records are restored, application of changes to Elasticsearch analyzers, and correction of indices after changes to base OSDU schemas or client schemas.
Disaster Recovery Scenario:
- All records in Storage (including underlying CosmosDB or FireStore or whatever) are brought back to RPO state.
- The Search index is not in sync yet with the restored Storage records, so Reindex All is executed.
- Reindex All should not be using the Schema get all schemas endpoint as that will retrieve every schema that has been defined in the installation which includes unused schemas and obsolete schemas and those may number in the thousands. Instead, Reindex All needs to use /query/kinds from Storage which will retrieve only those kinds actually in use in Storage.
- As Reindex All executes, the list of kinds is retrieved from Storage /query/kinds and iterated over, triggering a reindex on each individual kind known to Storage.