OS Core Lib Azure merge requestshttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests2022-12-15T17:57:07Zhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/268Upgrade com.azure:azure-core to latest version 1.34.02022-12-15T17:57:07ZThulasi Dass SubramanianUpgrade com.azure:azure-core to latest version 1.34.0- Upgrade com.azure:azure-core to latest version 1.34.0- Upgrade com.azure:azure-core to latest version 1.34.0M15 - Release 0.18Thulasi Dass SubramanianThulasi Dass Subramanianhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/270Cherry-pick 'Upgrade com.azure:azure-core to latest version 1.34.0' into rele...2022-12-15T17:58:21ZChad LeongCherry-pick 'Upgrade com.azure:azure-core to latest version 1.34.0' into release/0.18**Original MR**: !268
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !268
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/pipelines/new?ref=cherry-pick-for-268)M15 - Release 0.18David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/267Added CORS Feature Flag2022-12-22T15:07:26ZSrishti SharmaAdded CORS Feature Flag1. Tested CORS Feature Flag and updated code
2. Removed checkStyle issues
3. Updated appropriate variables names in Test class
4. PBI: 14514 E2E integration CORS1. Tested CORS Feature Flag and updated code
2. Removed checkStyle issues
3. Updated appropriate variables names in Test class
4. PBI: 14514 E2E integration CORSM16 - Release 0.19Vineeth Guna [Microsoft]Vineeth Guna [Microsoft]https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/269Adding reverted changes2023-01-12T23:53:24ZSrishti SharmaAdding reverted changes## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
Updating changes in MR:
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M16 - Release 0.19https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/271Update format of x-collaboration header2022-12-22T16:36:40ZMina OtgonboldUpdate format of x-collaboration header## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IB...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The format of x-collaboration header is:
"x-collaboration": "x-collaboration:id=1e1c4e74-3b9b-4b17-a0d5-67766558ec65,application=Test App"
## What is the new/expected behavior?
The format of x-collaboration header will be:
"x-collaboration": "id=1e1c4e74-3b9b-4b17-a0d5-67766558ec65,application=Test App"
## Have you added/updated Unit Tests and Integration Tests?
YesM16 - Release 0.19https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/273add async version of QUERY_ITEMS_API (CosmosAsyncClient)2023-01-12T10:38:20ZYurii Kondakovadd async version of QUERY_ITEMS_API (CosmosAsyncClient)Currently, using the existing client (Sync API), a certain number of extra (prefetch) queries are performed when retrieve records from the database page by page. This increases the consumption of Cosmos DB Requests Units (RUs).
To preven...Currently, using the existing client (Sync API), a certain number of extra (prefetch) queries are performed when retrieve records from the database page by page. This increases the consumption of Cosmos DB Requests Units (RUs).
To prevent this, an asynchronous client and an appropriate method for retrieving records from the database using this asynchronous client have been added.
[Issue](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/29)M16 - Release 0.19Yurii KondakovYurii Kondakovhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/275Add missing calculation of requestCharge for CosmosStore#queryItemsPageAsync ...2023-01-13T15:43:24ZYurii KondakovAdd missing calculation of requestCharge for CosmosStore#queryItemsPageAsync methodThis MR is an addition to just merged https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/273
It calculates requestCharge of the request (for logging and diagnostics purposes).This MR is an addition to just merged https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/273
It calculates requestCharge of the request (for logging and diagnostics purposes).M16 - Release 0.19Yurii KondakovYurii Kondakovhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/277M14 upgrade2023-05-31T10:43:02ZAnkur RawatM14 upgradeM14 upgradeM14 upgradeAnkur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/278increase version of os-core-common to 0.19.0-rc52023-07-13T03:28:14ZYauheni Lesnikauincrease version of os-core-common to 0.19.0-rc5Increase version of os-core-common to 0.19.0-rc5Increase version of os-core-common to 0.19.0-rc5M16 - Release 0.19Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/279Upgrade First Party Library Dependencies for Release 0.192023-02-18T07:31:15ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.19This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 51f8fc3f2b620753ef879f36d73d150e0124b690
Maven: 0.19.0-SNAPSHOT
```
| Maven Dependencies | _Root_ |
| ------------------------------------------------------- | ---------- |
| os-core-common | 0.19.0-rc5 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.24 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade-2
SHA: bb9add9128a68cfae975880264956f39c0d381a0
Maven: 0.19.0-SNAPSHOT
```
| Maven Dependencies | _Root_ |
| ------------------------------------------------------- | ------ |
| os-core-common | 0.19.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.24 |M16 - Release 0.19https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/281fix broken use of RequestMappingHandlerMapping in filter2024-01-02T08:34:57ZNeelesh Thakurfix broken use of RequestMappingHandlerMapping in filterAddresses issue: #30
On investigation, we found it's an [issue ](https://github.com/spring-projects/spring-boot/issues/28874) caused by Spring boot upgrade.
The fix applied in this MR are based on the resolution in the [comment](http...Addresses issue: #30
On investigation, we found it's an [issue ](https://github.com/spring-projects/spring-boot/issues/28874) caused by Spring boot upgrade.
The fix applied in this MR are based on the resolution in the [comment](https://github.com/spring-projects/spring-boot/issues/28874#issuecomment-1002008250).
We have also added cache on api handlers, so we don't have to do it every single request.
## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.M17 - Release 0.20https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/282Changes for using SPN token client and MSI token client instead of HTTP clien...2023-08-18T12:42:30ZAbhiram BondadaChanges for using SPN token client and MSI token client instead of HTTP client to fetch tokensTesting procedure: All documentation updated here
https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/_wiki/wikis/Open-Energy-Platform.wiki/2104/Testing-OSDU-services'-code-changes-against-a-healthy-instance
This is not ...Testing procedure: All documentation updated here
https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/_wiki/wikis/Open-Energy-Platform.wiki/2104/Testing-OSDU-services'-code-changes-against-a-healthy-instance
This is not a compatible branch that we are merging changes to
All commits in this branch are made on top of a previous version of core-lib (tag:v0.19.0-rc3). So, need to create another branch and MR.
This MR is just for saving changes, getting reviews and resolving the comments.M17 - Release 0.20Abhiram BondadaAbhiram Bondadahttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/283Changes for using SPN token client and MSI token client instead of HTTP clien...2023-08-18T12:42:28ZAbhiram BondadaChanges for using SPN token client and MSI token client instead of HTTP client to fetch tokensTesting procedure: All documentation updated here https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_wiki/wikis/Open-Energy-Platform.wiki/2104/Testing-OSDU-services'-code-changes-against-a-healthy-instance
This is not ...Testing procedure: All documentation updated here https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_wiki/wikis/Open-Energy-Platform.wiki/2104/Testing-OSDU-services'-code-changes-against-a-healthy-instance
This is not a compatible branch that we are merging changes to All commits in this branch are made on top of a previous version of core-lib (tag:v0.19.0-rc3). So, need to create another branch and MR. This MR is just for saving changes, getting reviews and resolving the comments.M17 - Release 0.20Abhiram BondadaAbhiram Bondadahttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/284Support for bulk patch2023-03-07T18:19:35ZAlok JoshiSupport for bulk patchAs part of [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/124), we are implementing a solution in Storage service. We want to use Cosmos's implicit patch document support to implement this feature in Storage ...As part of [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/124), we are implementing a solution in Storage service. We want to use Cosmos's implicit patch document support to implement this feature in Storage serivce. This change introduces the ability to do that via this library's methods. We are using CosmosContainer's bulk executor support.M17 - Release 0.20Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/285Add document id in bulk errros2023-03-10T16:30:38ZAlok JoshiAdd document id in bulk errrosUse cosmosItemOperation.getId() instead of cosmosItemOperation.getItem() to keep the message small
Return document id with errors when performing bulk operation
Will be useful in implementing [ADR](https://community.opengroup.org/osdu/...Use cosmosItemOperation.getId() instead of cosmosItemOperation.getItem() to keep the message small
Return document id with errors when performing bulk operation
Will be useful in implementing [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/124) to get info on recordIds in case of failureM17 - Release 0.20Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/286Added missing function for retrieving elastic versions for connected outer se...2023-03-15T04:39:01ZDeepa KumariAdded missing function for retrieving elastic versions for connected outer servicesThe connected outer services for azure do not contain elastic endpoints version. Analyzed the issue from Indexer and Search service and found that the implementation to getAllClustersSettings was missing inside ElasticClusterSettings cla...The connected outer services for azure do not contain elastic endpoints version. Analyzed the issue from Indexer and Search service and found that the implementation to getAllClustersSettings was missing inside ElasticClusterSettings class.
This MR is to add that behavior.M17 - Release 0.20Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/288add overload of queryItemsPage with CosmosQueryRequestOptions param2023-08-18T12:42:25ZNeelesh Thakuradd overload of queryItemsPage with CosmosQueryRequestOptions paramadds overload of queryItemsPage with CosmosQueryRequestOptions param. This param can be utilized by query/kind API to shorten continuationTokenadds overload of queryItemsPage with CosmosQueryRequestOptions param. This param can be utilized by query/kind API to shorten continuationTokenM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/287Validate data-partition-id in PartitionServiceClient2023-08-18T12:42:27ZAbhishek PatilValidate data-partition-id in PartitionServiceClient## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
Today we do not validate the data-partition-id because of which inbuilt URI class throws Illegal Argument Exception if an invalid data-partition-id is passed in headers.
For example, if a char `{` is passed into data-partition-id then Illegal Argument Exception is thrown which is ultimately converted into 500 error response by service. Ideally a 4XX response should have been returned.
This PR introduces a validation on data-partition-id which will help in responding back with 4XX response instead of 500.
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M17 - Release 0.20Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/289Patch per document2023-03-30T14:53:06ZAlok JoshiPatch per documentIt is a common use-case for end users to want to patch a list of documents with certain metadata attributes.
user makes a PATCH request:
```
{
"query": {
"ids": [
"record1",
"record2",
....It is a common use-case for end users to want to patch a list of documents with certain metadata attributes.
user makes a PATCH request:
```
{
"query": {
"ids": [
"record1",
"record2",
.
.
.
]
},
"ops": [
{
"op": "add",
"path": "/acl/viewers/-",
"value": "acl1"
},
{
"op": "add",
"path": "/acl/viewers/-",
"value": "acl2"
}
]
}
```
The expectation is that all records should have acl1 and acl2 as part of viewers acl. If acl1 or acl2 is already present for any records, it should be a no-op
Consider following scenario:
record1 has acl/viewers: ["acl1"]
record2 has acl/viewers: ["acl2]
This operation (or list of operations), if applied as-is, will create duplicate acl/viewers for certain records. To avoid this, record1 should only apply the second patch and record2 should only apply the first patch. Therefore, we can have a map input where cosmos patch operation can be different for each document. Corresponding mapping for Partition keys is also maintained
This is what the client (i.e. storage) is expected to generate when calling the bulk patch method.M17 - Release 0.20Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/290make redis host key and password key configurable2023-08-18T12:42:23ZMykyta Savchukmake redis host key and password key configurablemake redis host key and password key configurable in order to let storage use the premium redis instance which has more spare capacity to reduce the load on the standard redis instancemake redis host key and password key configurable in order to let storage use the premium redis instance which has more spare capacity to reduce the load on the standard redis instanceM18 - Release 0.21Mykyta SavchukMykyta Savchuk