OS Core Lib Azure merge requestshttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests2023-08-18T12:43:05Zhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/211Bulk create using Cosmos Client2023-08-18T12:43:05ZAnkur RawatBulk create using Cosmos ClientAdded a method to bulk create the records using CosmosClient. CosmosClient can be created either with MSI or Cosmos-primary Key.Added a method to bulk create the records using CosmosClient. CosmosClient can be created either with MSI or Cosmos-primary Key.M13 - Release 0.16Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/23BugFix: moving client creation logic out of constructor2023-08-18T12:46:37ZAman VermaBugFix: moving client creation logic out of constructor## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES/NO] I have updated the documentation accordingly. NA
* [YES/NO/NA]...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES/NO] I have updated the documentation accordingly. NA
* [YES/NO/NA] I have added tests to cover my changes. NA
* [YES/NO/NA] All new and existing tests passed. YES
* [YES/NO/NA] My code follows the code style of this project. YES
* [YES/NO/NA] I ran lint checks locally prior to submission. YES
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue. -->
Description:
Currently, logic to initialize `BlobServiceClient` is inside the constructor. But it relies on `DefaultAzureCredential` bean which has been `Autowired` and at run time we get `NullRefException` as the autowired bean is yet to be picked
FIX: Tagging the constructor with `Autowired` tag so that other autowired components can be passed as parameters.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
TESTING: Tested the new package version locally with schema service. All ITs passing.
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Other information
-------------------------------------
<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->
FYI @kibattul , @polavishnuM1 - Release 0.1Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/104Blobstore Retry Configuration2023-08-18T12:44:43ZRonak SakhujaBlobstore Retry Configuration## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the documentation accordingly.
* [YES] I have added tests ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
BlobStore offers option for configuring various attributes for retry policy. For any service who wish not to use the default values, and want to provide custom values for the mentioned attributes, they will be able to after this change.
High level design:
![image](/uploads/5d300661ada45b326a9c0043da4f9b89/image.png)
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
Change details:
Variables that can be added to application.properties are:
azure.blobstore.maxTries
azure.blobstore.tryTimeoutInSeconds
azure.blobstore.retryDelayInMs
azure.blobstore.maxRetryDelayInMs
azure.blobstore.retryPolicyType
azure.blobstore.secondaryHost
## Test coverage:
------------------
All changes are covered.
![image](/uploads/93e03dc4635021ba219dbefad4cdbca1/image.png)
## Does this introduce a breaking change?
-------------------------------------
- NO
## Pending items
----------------
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M7 - Release 0.10https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/41Azure SDKs Upgrade & Dependency Management2023-08-18T12:46:13ZAbhishek PatilAzure SDKs Upgrade & Dependency Management## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [NA] I have added tests ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NO] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
- Azure SDKs upgrade.
- Managed dependencies which are common across all OSDU services.
- HLD [link](https://microsoft-my.sharepoint.com/:w:/p/abpatil/EVbuX6l-7qJAhrjDKyk4PT8BeewI_iZg0mPAwb2WIBhyBg?e=XPJTm6)
Issue: https://dev.azure.com/msazure/One/_workitems/edit/8708897<!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
- Azure SDKs are upgraded to latest stable version.
- Managed dependencies (azure spring boot starters, reactor-bom, netty-bom etc.) added in POM. These can be inherited by each OSDU service and hence a consistent set of versions for these dependencies will be used across all OSDU services.
## Test coverage:
------------------
Not applicable<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
Raise MRs for all OSDU services to use this version of core-lib-azure.<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
None<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/205ApplicationInsights flag integration2023-08-18T12:43:07ZAnubhav AronApplicationInsights flag integration**Functional Problem statement:**
In OAK, Application Insights is disabled and Logs that are emitted to Application Insights are getting dropped.
**Expected Behavior:**
Integrate feature flag to decide destination [Geneva or Applicat...**Functional Problem statement:**
In OAK, Application Insights is disabled and Logs that are emitted to Application Insights are getting dropped.
**Expected Behavior:**
Integrate feature flag to decide destination [Geneva or ApplicationInsights]. In OAK, logs will send to Geneva In OSDU, logs dependency to be send to ApplicationInsights.
**Technical Problem statement:**
In case of applicationInsights, for every logger call, a new applicationInsight client is getting created with same configuration.
**Expected Behavior:**
If application Insights is enabled, only one Application client will be created.M12 - Release 0.15Anubhav AronAnubhav Aronhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/60Add user ID to MDC map in Slf4jMDCFilter2023-08-18T12:45:40ZKelly DomicoAdd user ID to MDC map in Slf4jMDCFilterAdd user ID to MDC map in Slf4jMDCFilter so it can be added to the customDimensions object in App Insights tables. By default, we are using the subject for the user ID.
Additional add util class for unwrapping token to get claims.Add user ID to MDC map in Slf4jMDCFilter so it can be added to the customDimensions object in App Insights tables. By default, we are using the subject for the user ID.
Additional add util class for unwrapping token to get claims.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/228Add user delegation sas method to generate signed url for Datalake storage2023-08-18T12:42:54Zharshit aggarwalAdd user delegation sas method to generate signed url for Datalake storage## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M14 - Release 0.17harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/56Add trace and dependency logging to KeyVaultFacade2023-08-18T12:45:47ZKelly DomicoAdd trace and dependency logging to KeyVaultFacadeAdd trace and dependency logging (via CoreLogger) to KeyVaultFacade.Add trace and dependency logging (via CoreLogger) to KeyVaultFacade.M1 - Release 0.1Kishore BattulaKishore Battulahttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/61Add to customDimensions from MDC for logged requests2023-08-18T12:45:39ZKelly DomicoAdd to customDimensions from MDC for logged requestsCurrently, the MDC map key-value pairs are not included in logged requests. Request and exception logging is provided out-of-the-box with the App Insights and Spring Boot web JARs.
This MR adds a TelemetryInitializer for web requests th...Currently, the MDC map key-value pairs are not included in logged requests. Request and exception logging is provided out-of-the-box with the App Insights and Spring Boot web JARs.
This MR adds a TelemetryInitializer for web requests that will allow adding custom dimensions for requests. It copies the MDC properties map to the Telemetry object's properties map. The properties will then show up in customDimensions in the requests table.M1 - Release 0.1Kishore BattulaVibhuti Sharma [Microsoft]Kishore Battulahttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/30adds queryItemsWithContinuationToken to CosmosStore2020-10-02T15:11:03ZAliaksei Darafeyeuadds queryItemsWithContinuationToken to CosmosStoreNitin-slbNeelesh ThakurPavel BachylaAlok JoshiDuvelis CaraoNitin-slbhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/65adds CryptographyClientFactory andEventGridManagerFactory2023-08-18T12:45:32ZAliaksei Darafeyeuadds CryptographyClientFactory andEventGridManagerFactoryM1 - Release 0.1Neelesh ThakurAlok JoshiNeelesh Thakurhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/36adds ability use own ElasticCredentialsCache impl2023-08-18T12:46:18ZAliaksei Darafeyeuadds ability use own ElasticCredentialsCache implM1 - Release 0.1Neelesh ThakurAlok JoshiDuvelis CaraoNeelesh Thakurhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/288add overload of queryItemsPage with CosmosQueryRequestOptions param2023-08-18T12:42:25ZNeelesh Thakuradd overload of queryItemsPage with CosmosQueryRequestOptions paramadds overload of queryItemsPage with CosmosQueryRequestOptions param. This param can be utilized by query/kind API to shorten continuationTokenadds overload of queryItemsPage with CosmosQueryRequestOptions param. This param can be utilized by query/kind API to shorten continuationTokenM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/186Add option to use MSI for Authentication to Azure Resources2023-08-18T12:43:26Zharshit aggarwalAdd option to use MSI for Authentication to Azure ResourcesChanges in the MR are focused on adding option to use Managed Identity Credentials while authenticating to Azure resourcesChanges in the MR are focused on adding option to use Managed Identity Credentials while authenticating to Azure resourcesM11 - Release 0.14harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/203Add option to read AAD variables from env2023-08-18T12:43:08Zharshit aggarwalAdd option to read AAD variables from env## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->
AddM12 - Release 0.15harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/238add missing RU on dependency logs for QUERY_ITEMS API2023-08-18T12:42:47ZNeelesh Thakuradd missing RU on dependency logs for QUERY_ITEMS API## All Submissions:
add missing RU on dependency logs for QUERY_ITEMS API
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the...## All Submissions:
add missing RU on dependency logs for QUERY_ITEMS API
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## Does this introduce a breaking change?
-------------------------------------
- [NO]M14 - Release 0.17https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/275Add missing calculation of requestCharge for CosmosStore#queryItemsPageAsync ...2023-01-13T15:43:24ZYurii KondakovAdd missing calculation of requestCharge for CosmosStore#queryItemsPageAsync methodThis MR is an addition to just merged https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/273
It calculates requestCharge of the request (for logging and diagnostics purposes).This MR is an addition to just merged https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/273
It calculates requestCharge of the request (for logging and diagnostics purposes).M16 - Release 0.19Yurii KondakovYurii Kondakovhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/195Add Method for generating SAS token in blobstore2023-08-18T12:43:17Zharshit aggarwalAdd Method for generating SAS token in blobstoreM11 - Release 0.14harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/208Add `keyVaultUrl` in PartitionInfo for Secret service2023-02-04T05:36:55ZDmitrii Novikov (EPAM)Add `keyVaultUrl` in PartitionInfo for Secret service## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES/NO/NA] I have added...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes. - TBD - Opened up for initial review
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
Secret service need new property `keyVaultUrl` in PartitionInfo
## Does this introduce a breaking change?
-------------------------------------
- [NO]Dmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/212Adding validation for private links2023-08-18T12:43:03ZHarsheet ShahAdding validation for private links## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? Yes
* [YES/NO] I have updated the documentation accordingly. Yes
* [YES/NO/NA...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? Yes
* [YES/NO] I have updated the documentation accordingly. Yes
* [YES/NO/NA] I have added tests to cover my changes. Yes
* [YES/NO/NA] All new and existing tests passed. Yes
* [YES/NO/NA] My code follows the code style of this project. Yes
* [YES/NO/NA] I ran lint checks locally prior to submission. Yes
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
Validation for private links
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO] Yes
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M13 - Release 0.16Harsheet ShahHarsheet Shah