Post-Elastic 7.x Update, Search query record counts are clipped at 10k records
Summary
The default Elastic search setting of 10k records (documents) returned for large queries is in force after the Elastic 7.x upgrade for OSDU. This is an undesirable change in behavior compared to the previous Elastic settings. The expected behavior is to return the "limit" number of records (e.g. 500) but the record count should not be capped at 10k. It should be the actual record count.
Details
Scenario 1: kind query with more than 10k records. Actual record count in Storage = 24,401, using Search query (Python client) search.query(kind=,limit=1).total_count will only return 10,000 records. And, of course, the 1 result specified by the limit=1 parameter. The previous implementation would have returned 24,401 for the "total_count".