Azure CosmosDb saturation/throttling when records reach too many versions
Context:
In Jul 2023 an issue has been raised about Azure Cosmos DB saturation #178 (closed)\\ Two MRs were proposed and merged as a solution for the issue:
- Restriction on records number, under Feature Flag (M25) !834 (merged)
- New Deletion versions API (M25) - !862 (merged)
Than (Aug 2024) an issue "Storage record restriction of gcs paths array causes ingestion to fail" was created #254\\ And MR for this issue was merged - !923 (merged) - actually a revert for restriction on number version MR(834)
That is, at the moment, the restriction on the number of versions has been removed, and the problem with database throttling can only be mitigated if the end user independently calls the Deletion Version API.
The main reason why the restriction on the number of versions was removed is that the service response when the restriction was triggered only contained a list of skipped records and did not contain the reason for skipping.
There is a proposal to return the restriction on the number of versions for createUpdateRecords functionality, which will be implemented as before under the feature flag, as well as to add to CreateUpdateRecordsResponse a list of reasons (errors) why the records were skipped and not added/updated to the database.