OS Core Lib Azure merge requestshttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests2023-08-18T12:45:31Zhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/66Adding Cosmos Bulk Insert2023-08-18T12:45:31ZJasonAdding Cosmos Bulk Insert## All Submissions:
-------------------------------------
* [**YES**/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [**YES**/NO] I have updated the documentation accordingly.
* [**YES**/N...## All Submissions:
-------------------------------------
* [**YES**/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [**YES**/NO] I have updated the documentation accordingly.
* [**YES**/NO/NA] I have added tests to cover my changes.
* [**YES**/NO/NA] All new and existing tests passed.
* [**YES**/NO/NA] My code follows the code style of this project.
* [YES/NO/**NA**] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
Our current CosmosStore does not perform well with large amounts of writes. This MR introduces the DocumentBulkExecutor package, which is specifically designed by Microsoft to perform large writes. Initial tests have shown large performance gains when using this new package for writing large volumes of records to storage.
High level design: Mirrors the pattern of CosmosClient. Has a class responsible for creating the client, and another class (ComsosStoreBulkOperations) that exposes the APIs from the package.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. --> N/A
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details: New constants are defined in CosmosBulkExecutorConfiguration file. Default values for these constants were set based on the default values specified in the SDK itself. Specifying the number of RUs to allocate for Cosmos bulk executor does not have a default, so I specified the default as 4000 (our default environment configuration has a total of 12000 RUs). Other than that it should hopefully be straight forward as it mirror the pattern laid out by CosmosClient.
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
Unit test coverage mirrors that of original CosmosStore implementation. Tests for null partition db, empty partition id, and that the client is being cached correctly.
## Does this introduce a breaking change?
-------------------------------------
- [YES/**NO**]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
No, not in this repo. The new code will be used first in storage service.
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1JasonJasonhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/67Added tests for AuthUtils2023-08-18T12:45:29ZKelly DomicoAdded tests for AuthUtilsConverted AuthUtils to Spring components
Added tests for AuthUtilsConverted AuthUtils to Spring components
Added tests for AuthUtilsM1 - Release 0.1Kishore BattulaKishore Battulahttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/68Fixing UT for Service Bus Client2023-08-18T12:45:27ZKomal MakkarFixing UT for Service Bus Client## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
The UT
> should_return_client_when_partition_valid
was failing. Fixed the same by refactoring.
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
Increased.
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
None.
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1Hema Vishnu Pola [Microsoft]Hema Vishnu Pola [Microsoft]https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/69Context Aware Concurrency2023-08-18T12:45:25ZVibhuti Sharma [Microsoft]Context Aware Concurrency## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NO] I have u...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NO] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Create custom classes for concurrency which extend logic from classes in java.util.concurrency package but ensure that the MDC and request context is copied from main thread to child threads.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
To handle concurrency in our services, we make use of java.util.concurrency package. This package contains classes like `ExecutorService` which handle the thread pool for us and execute Runnable objects in new threads. The implementation is such that thread context is not copied from main thread to child threads, and we lose the MDC context map and other request attributes in the child threads.
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
Added unit tests for `CustomExecutors` and `CustomThreadPoolExecutorUtil`.
## Does this introduce a breaking change?
-------------------------------------
- NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
Upcoming MR in storage service to add CustomThreadPoolFactory - https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/126
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change.M1 - Release 0.1Vibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/71Container creation and deletion in BlobStore2023-08-18T12:45:24ZAalekh JainContainer creation and deletion in BlobStore## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NO] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
**Current behaviour**: There is no method in BlobStore to create and delete container in azure storage account which. There has been a need for this in the Ingestion service, which requires creation and deletion of container on the fly
**Expected behaviour**: Will be able to call methods directly from BlobStore to either create / delete containers by providing their container id.
This PR adds the method for both the creation and deletion of containers in azure storage account with 100% code coverage for the newly added methods by adding appropriate UTs as well.
## Test coverage:
UTs are added for the newly introduced methods. All UTs pass.
## Does this introduce a breaking change?
- [NO]
## Reviewer request
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/72Fetching Azure subscription id from keyvault2023-08-18T12:45:21Zharshit aggarwalFetching Azure subscription id from keyvaultThis MR is making changes to fetch Azure subscription Id from Azure Keyvault
Since this property is not partition sensitive, therefore it should not be part of partition specific properties and we can directly read it from keyvault like...This MR is making changes to fetch Azure subscription Id from Azure Keyvault
Since this property is not partition sensitive, therefore it should not be part of partition specific properties and we can directly read it from keyvault like servicePrincipleAppIdConfig
Relates with https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/65
and https://community.opengroup.org/osdu/platform/system/register/-/merge_requests/64M1 - Release 0.1Neelesh ThakurAlok JoshiNeelesh Thakurhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/73Fixing Bulk Executor Configurations2023-08-18T12:45:20ZJasonFixing Bulk Executor Configurations## All Submissions:
-------------------------------------
* [**YES**/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [**YES**/NO] I have updated the documentation accordingly.
* [YES/NO/**...## All Submissions:
-------------------------------------
* [**YES**/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [**YES**/NO] I have updated the documentation accordingly.
* [YES/NO/**NA**] I have added tests to cover my changes.
* [YES/NO/**NA**] All new and existing tests passed.
* [**YES**/NO/NA] My code follows the code style of this project.
* [YES/NO/**NA**] I ran lint checks locally prior to submission.
The issues:
- Cosmos Bulk Executor configuration is currently using System.getProperty when it needs to be using System.getenv (properties are properties passed in when running the .jar, env references environment variables, which is what we want
- Cosmos Bulk Executor is currently creating a number of threads = 100 * # of available processors. We want to change this to a constant number to be conscious of memory usage by bulk executor.
The fixes:
- Switched to System.getenv
- Switched to a constant of 100 threads maxM1 - Release 0.1JasonJasonhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/74policy service feature flag2023-08-18T12:45:18ZAlok Joshipolicy service feature flagAdd policy service feature flag by adding 'policy-service-enabled' field for partitions. The field will act as a feature flag to use Policy service in integration with Storage and Search for certain partitions.Add policy service feature flag by adding 'policy-service-enabled' field for partitions. The field will act as a feature flag to use Policy service in integration with Storage and Search for certain partitions.M1 - Release 0.1Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/76log exception for not found record on read request2023-08-18T12:45:15ZNeelesh Thakurlog exception for not found record on read requestM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/79Event Grid Topic dynamic addition2023-08-18T12:45:13ZKomal MakkarEvent Grid Topic dynamic addition## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] Does the MR contain pipeline/ helm chart related changes?
* [YES] I have...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] Does the MR contain pipeline/ helm chart related changes?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
__Existing Behaviour__: The Event Grid Topics were handled as an enum. This will force the consumer of the facade to contribute to the facade on the creation of a new topic.
__Updated Behaviour__
Event Grid Facade has the ability to pick up any topic which confides to the naming convention as discussed in (this wiki)[https://community.opengroup.org/osdu/platform/system/notification/-/merge_requests/62].
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
Jacoco is absent for the repo.
![image](/uploads/a26a92ad4fd4b5321192e2313b39e7c3/image.png)
![image](/uploads/5171808543dc801f07219eaf4655b7fa/image.png)
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
Storage service has to react to this change.
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M4 - Release 0.7Kishore BattulaKishore Battulahttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/81Fix authorization header bug in PartitionServiceClient2023-08-18T12:45:11ZRostyslav Matushkin (SLB)Fix authorization header bug in PartitionServiceClient## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NA] I have updated the documentation accordingly.
* [YES] I have added tests ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NA] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
__Existing behaviour__: Each invocation of the method `org.opengroup.osdu.azure.partition.PartitionServiceClient:getServiceClient` overwrites the existing Authorization header in DpsHeaders, the token is overriden by result from the method `org.opengroup.osdu.azure.util.AzureServicePrincipleTokenService:getAuthorizationToken`. This is an erroneous behavior that overwrites the token of the client accessing the service.
__New behaviour__: Each invocation of the method `org.opengroup.osdu.azure.partition.PartitionServiceClient:getServiceClient` creates a new instance of DpsHeaders and puts there the Authorization header by result from the method `org.opengroup.osdu.azure.util.AzureServicePrincipleTokenService:getAuthorizationToken`. The new instance of DpsHeaders is used for `org.opengroup.osdu.core.common.partition.PartitionService` but the old instance remains unchanged having the correct token.
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
## Test coverage:
------------------
![Screenshot_2021-03-02_at_16.40.03](/uploads/564c5c0d29a803f515d135720892a08f/Screenshot_2021-03-02_at_16.40.03.png)
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
[Entitlements service](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements) requires this change to fix authorization logic.
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_M4 - Release 0.7Rostyslav Matushkin (SLB)Rostyslav Matushkin (SLB)https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/85Fixing logs in BlobStore2023-08-18T12:45:10ZKrishna Nikhil VedurumudiFixing logs in BlobStore## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
The logging was broken.
Actual - `Done uploading file content to %s`
Expected - `Done uploading file content to {filePath}`
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/86Fix CVE security vulnerabilities, upgrade to latest core-common2023-08-18T12:45:08ZAlok JoshiFix CVE security vulnerabilities, upgrade to latest core-commonUpdate to use latest core-common release candidate, which has security vulnerabilities addressed.
Please see [this MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/75)Update to use latest core-common release candidate, which has security vulnerabilities addressed.
Please see [this MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/75)M5 - Release 0.8Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/87Using Expire After Write Strategy for caching2023-08-18T12:45:06ZKrishna Nikhil VedurumudiUsing Expire After Write Strategy for caching## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
The client connections that we manage for Cosmos and Blob Store should be refreshed periodically. Using After Access expiry strategy will not let the stale connections to expire.
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M6 - Release 0.9https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/88simple-xml jar fix2023-08-18T12:45:04ZLarissa Pereirasimple-xml jar fixFix for the vulnerable simple-xml jar that was identified by our internal scansFix for the vulnerable simple-xml jar that was identified by our internal scansM6 - Release 0.9https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/90Fix security vulnerabilities2023-08-18T12:45:03ZRostislav Vatolinvatolinrp@gmail.comFix security vulnerabilitiesMore details: https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/8More details: https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/8M6 - Release 0.9https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/91Replace CosmosCache with Map2023-08-18T12:45:01ZKrishna Nikhil VedurumudiReplace CosmosCache with Map## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES] I have added...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
As per the recommendation from Cosmos team, there should be single client of Cosmos during the entire life cycle of the application.
The cosmos clients are heavy in nature and can eat up significant amount of memory. Using a cache is leading to two issues.
1. Expiry of existing clients causes cache eviction which is leading to connection leaks.
2. It is anyway not required, given that whatever Cosmos GoneExceptions are raised by the Cosmos are handled by the SDK itself.
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->
Before fix, during scale up - the number of GoneExceptions were high.
![Before_fix](/uploads/1cb6196d3c6c195d3f11d32b3f33b4bc/Before_fix.PNG)
### After fix, during scale up - no GoneExceptions
![After_fix](/uploads/725bfb17da48a467ff7f936bbb73496f/After_fix.PNG)
### Logs that confirm Cosmos client per partition is created only once.
![Cosmos_Created_only_once](/uploads/3665f5bb55d6ee1106bef9a7b5728538/Cosmos_Created_only_once.PNG)
### Response status with scale up
![5xx_while_scale_up](/uploads/d40a1bdb6bd184a0dc3d63673310e3a8/5xx_while_scale_up.PNG)M6 - Release 0.9Krishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/92Add configuration properties for event-grid & cryptography factories, partiti...2023-08-18T12:44:59ZNeelesh ThakurAdd configuration properties for event-grid & cryptography factories, partition service initialization fails otherwiseAzure resources are partition aware, they will try to initialize partition service client when they are integrated with Core services. In case of Partition service, it's not desired behavior. Partition service do rely on core-lib-azure f...Azure resources are partition aware, they will try to initialize partition service client when they are integrated with Core services. In case of Partition service, it's not desired behavior. Partition service do rely on core-lib-azure for common azure functionality so we cannot take out the core-lib-azure reference from service. These config property will allow us to disable injection of these factories.M6 - Release 0.9https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/93Managing common properties from core-lib-azure2023-08-18T12:44:57ZAbhishek PatilManaging common properties from core-lib-azure## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [NA] I have added tests ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
There are various configuration properties which are common across all OSDU services. Any update/addition to these common properties require change in each and every service using it.
High level design:
https://microsoft.sharepoint.com/:w:/t/EnergyDataPlatform/EckjpjanywtMhRIzwgHotXABpRuHueQl1GSmdySPn0tNnw?e=XfZIlw
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
https://dev.azure.com/msazure/One/_sprints/taskboard/Azure%20Global%20Engineering%20Energy%20Team/One/AGE/Cobalt/May?workitem=9872385
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
Adding common.properties file which will contain all the configuration properties which are common across all OSDU services. Services will use this common.properties along with their own application.properties.
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M7 - Release 0.10Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/95Adding the maven release-candidiate script to this library's pipeline2023-08-18T12:44:55ZDavid Diederichd.diederich@opengroup.orgAdding the maven release-candidiate script to this library's pipelineM6 - Release 0.9David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.org