Fixing Bulk Executor Configurations (!73) · Merge requests · OSDU Software / OSDU Data Platform / System / Lib / cloud / azure / OS Core Lib Azure · GitLab

Starting on April 8th, The Open Group will require Two Factor Authentication for all GitLab users. For instructions on setting up 2FA, see the Knowledge Article or GitLab Documentation. Note in particular that any HTTPS clones will require the use of a Personal Access Token (rather than a user’s password) once 2FA is enabled.

Jason requested to merge cosmos-bulk-executor-fixes into master Jan 20, 2021

All Submissions:

[YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
[YES/NO] I have updated the documentation accordingly.
[YES/NO/NA] I have added tests to cover my changes.
[YES/NO/NA] All new and existing tests passed.
[YES/NO/NA] My code follows the code style of this project.
[YES/NO/NA] I ran lint checks locally prior to submission.

The issues:

Cosmos Bulk Executor configuration is currently using System.getProperty when it needs to be using System.getenv (properties are properties passed in when running the .jar, env references environment variables, which is what we want
Cosmos Bulk Executor is currently creating a number of threads = 100 * # of available processors. We want to change this to a constant number to be conscious of memory usage by bulk executor.

The fixes:

Switched to System.getenv
Switched to a constant of 100 threads max