Adding Cosmos Bulk Insert
All Submissions:
- [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
- [YES/NO] I have updated the documentation accordingly.
- [YES/NO/NA] I have added tests to cover my changes.
- [YES/NO/NA] All new and existing tests passed.
- [YES/NO/NA] My code follows the code style of this project.
- [YES/NO/NA] I ran lint checks locally prior to submission.
What is the issue or story related to the change?
Our current CosmosStore does not perform well with large amounts of writes. This MR introduces the DocumentBulkExecutor package, which is specifically designed by Microsoft to perform large writes. Initial tests have shown large performance gains when using this new package for writing large volumes of records to storage.
High level design: Mirrors the pattern of CosmosClient. Has a class responsible for creating the client, and another class (ComsosStoreBulkOperations) that exposes the APIs from the package.
Issue: N/A
Change details: New constants are defined in CosmosBulkExecutorConfiguration file. Default values for these constants were set based on the default values specified in the SDK itself. Specifying the number of RUs to allocate for Cosmos bulk executor does not have a default, so I specified the default as 4000 (our default environment configuration has a total of 12000 RUs). Other than that it should hopefully be straight forward as it mirror the pattern laid out by CosmosClient.
Test coverage:
Unit test coverage mirrors that of original CosmosStore implementation. Tests for null partition db, empty partition id, and that the client is being cached correctly.
Does this introduce a breaking change?
- [YES/NO]
Pending items
No, not in this repo. The new code will be used first in storage service.
Reviewer request
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR -- Coding design: <Reviewer1> -- Backward Compatibility: <Reviewer2> -- Feature Logic: <Logic design> -- <Any other context mention here> OR -- <Component 1>: <Reviewer1> -- <CosmosDB>: <Reviewer2> -- <ServiceBus> <Reviewer3> -- <Mention any other component and owner>