OS Core Lib Azure merge requestshttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests2022-12-08T17:28:59Zhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/264Cherry-pick 'Expose Collaboration Context in message for new topic (recordsev...2022-12-08T17:28:59ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Expose Collaboration Context in message for new topic (recordsevent topic)' into release/0.18**Original MR**: !257
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !257
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/pipelines/new?ref=cherry-pick-for-257)M15 - Release 0.18David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/263Upgrade First Party Library Dependencies for Release 0.182022-12-08T08:36:39ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.18This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: release/0.18
SHA: eb79ab42caf333e3acc0756f6c5803f474ff21df
Maven: 0.18.0-SNAPSHOT
```
| Maven Dependencies | _Root_ |
| ------------------------------------------------------- | ---------- |
| os-core-common | 0.18.0-rc3 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.4 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: c3d208a06ab1865292173916fa9decbe7b83e0c0
Maven: 0.19.0-SNAPSHOT
```
| Maven Dependencies | _Root_ |
| ------------------------------------------------------- | ------ |
| os-core-common | 0.18.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.4 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 |
# Why is this a direct patch?
The default branch has already moved on to os-core-common `0.19.0-rc1`.
So, there's no benefit in merging this to the default branch (and it would undo work already started towards the next release), but we need it here for the 0.18 release.M15 - Release 0.18David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/262Update FOSSA NOTICE2022-12-04T08:35:22ZDavid Diederichd.diederich@opengroup.orgUpdate FOSSA NOTICEThis MR updates the attribution file for the project (also known as the `NOTICE` file).
It is important to keep this up to date to satisfy legal requirements of dependency licenses.
We use FOSSA as the tool to scan for and detect these ...This MR updates the attribution file for the project (also known as the `NOTICE` file).
It is important to keep this up to date to satisfy legal requirements of dependency licenses.
We use FOSSA as the tool to scan for and detect these changes.M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/261CORS | Send back null headers from all services so that istio CORS policy hon...2023-08-18T12:42:33ZSrishti SharmaCORS | Send back null headers from all services so that istio CORS policy honored* Feature: Implement CORS
* We update CORS policy in istio
* Currently all services send back hardcoded response headers due to which istio's CORS is not implemented.
* Created a new class in core-lib-azure which implements Filter. It se...* Feature: Implement CORS
* We update CORS policy in istio
* Currently all services send back hardcoded response headers due to which istio's CORS is not implemented.
* Created a new class in core-lib-azure which implements Filter. It sends back null headers from all services so that istio CORS policy honored.
**Response: 403 - CORS Error for different origin - Expected**
![image](/uploads/e457c1bbe79b6124980e6bb7af8cb461/image.png)
![image](/uploads/3d7b122e8aa984ea401a6178e3a6960a/image.png)
**Console:** Access to XMLHttpRequest at 'https://mstest3338.oep.ppe.azure-int.net/api/file/v2/files/uploadURL' from origin 'https://www.test-cors.org' has been blocked by CORS policy: **Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource**.
**Response: 200 response for valid origin – Expected Result** \*\*Updated istio CORS to allow origin https://www.test-cors.org
![image](/uploads/1fff10bde7f547be73e7c271d3c04fa0/image.png)M16 - Release 0.19https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/260Remove private links code2022-12-22T15:07:54ZHarsheet ShahRemove private links code## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M16 - Release 0.19Harsheet ShahHarsheet Shahhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/259Merge branch 'ankurrawat/ErrorMessageFix' into 'release/0.17'2022-12-22T15:08:57ZAnkur RawatMerge branch 'ankurrawat/ErrorMessageFix' into 'release/0.17'4xx error handling for bulkUpdate Operation4xx error handling for bulkUpdate OperationM15 - Release 0.18Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/258Handling 4xx in CosmosBulkOperations2022-12-22T18:58:04ZAnkur RawatHandling 4xx in CosmosBulkOperationsWhen CosmosDb throws 4xx, it is logged and not thrown. With this API returns 201 in response.
Catching exception from CosmosDB and throwing it to return correct response.When CosmosDb throws 4xx, it is logged and not thrown. With this API returns 201 in response.
Catching exception from CosmosDB and throwing it to return correct response.Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/257Expose Collaboration Context in message for new topic (recordsevent topic)2022-12-08T08:03:44ZMina OtgonboldExpose Collaboration Context in message for new topic (recordsevent topic)## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NA] I have updated the documentation accordingly.
* [YES] I have added tests ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NA] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
[ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/149)
Change details:
The message for the new topic (recordsevent) contains collaboration context if provided
## Test coverage:
------------------
Added unit test to verify that the new message contains collaboration context if provided
## Does this introduce a breaking change?
-------------------------------------
- [NO]M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/256Coreservicesupgradev12022-11-15T16:30:48ZNishant VidyasagarCoreservicesupgradev1Adding path matching strategy so, that other dependent services can use this.Adding path matching strategy so, that other dependent services can use this.M15 - Release 0.18Nishant VidyasagarNishant Vidyasagarhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/255Added a regex filter to filter out a particular pattern of msgs received from...2023-08-18T12:42:35ZNaga Aneesh MylavarapuAdded a regex filter to filter out a particular pattern of msgs received from core message receiver## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_sprints/taskboard/Energy%20-%20Explorers/Open%20Energy%20Platform/Oct%20'22/Oct%20'22%20Sprint%203?workitem=7737
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details: Added a regex filter to filter out messages of a particular pattern received from CoreMessageReceiver. Added a threshold filter to ensure that messages are removed only if their level is info or below.
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO] No
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M15 - Release 0.18Naga Aneesh MylavarapuNaga Aneesh Mylavarapuhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/254coreservices upgrade containing blob storage and file datalake2022-11-15T16:31:19ZNishant Vidyasagarcoreservices upgrade containing blob storage and file datalake## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] All new and existing tests passed.
* [YES] My code follows the code styl...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
For the Azure Blob Storage, we are using the SDK version:- 12.14.2(2021-11-10).The latest SDK version is:- 12.19.1 (2022-09-12) azure-blob-storage SDK version, So, upgraded the same. Since, Intermittent 500s were coming due to azure blob throwing 400, so needed to upgrade the Azure Blob Storage. Upgrading the same added some more dependencies.
## Test coverage:
------------------
ran Service IT's locallyM15 - Release 0.18Nishant VidyasagarNishant Vidyasagarhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/250Update src/main/java/org/opengroup/osdu/azure/publisherFacade/PubsubConfigura...2023-08-18T12:42:37ZNikhil Singh[MicroSoft]Update src/main/java/org/opengroup/osdu/azure/publisherFacade/PubsubConfiguration.java## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/249increase version of os-core-common to 0.18.0-rc32023-08-18T12:42:38ZYauheni Lesnikauincrease version of os-core-common to 0.18.0-rc3increase version of os-core-common to 0.18.0-rc3 to fix CVE-2022-42004 whitesource security vulnerability for the jackson-databind library
Related MR: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_...increase version of os-core-common to 0.18.0-rc3 to fix CVE-2022-42004 whitesource security vulnerability for the jackson-databind library
Related MR: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/182M15 - Release 0.18Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/248increase version of os-core-common to 0.18.0-rc22023-08-18T12:42:40ZYauheni Lesnikauincrease version of os-core-common to 0.18.0-rc2Increase version of os-core-common with to 0.18.0-rc2 fix of whitesource security vulnerabilities.
Merged MR of os-core-common: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/179
Fix white...Increase version of os-core-common with to 0.18.0-rc2 fix of whitesource security vulnerabilities.
Merged MR of os-core-common: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/179
Fix whitesource high severity security vulnerabilities for next dependencies:
snakeyaml-1.26.jar - CVE-2022-25857
jackson-databind-2.13.2 - CVE-2022-42004M15 - Release 0.18Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/247Upgrade First Party Library Dependencies for Release 0.172022-10-01T04:31:30ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.17This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: b94dd862b8fe279693563b50d9cd9a516c6389ee
Maven: 0.17.0-SNAPSHOT
```
| Maven Dependencies | _Root_ |
| ------------------------------------------------------- | ---------- |
| os-core-common | 0.17.0-rc2 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.2.2 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: a055c62503f1ee1c204e851e31e23a330cbe5c58
Maven: 0.17.0-SNAPSHOT
```
| Maven Dependencies | _Root_ |
| ------------------------------------------------------- | -------- |
| os-core-common | 0.17.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.2.2 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 |M14 - Release 0.17https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/245update NOTICE to pass fossa-check-notice in master2023-08-18T12:42:42ZMina Otgonboldupdate NOTICE to pass fossa-check-notice in masterThis MR updates NOTICE to pass the master pipeline's fossa-check-notice step in the scan stage: https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/pipelines/138176
## All Submissions:
--------------...This MR updates NOTICE to pass the master pipeline's fossa-check-notice step in the scan stage: https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/pipelines/138176
## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NA] I ran lint checks locally prior to submission.
## Does this introduce a breaking change?
-------------------------------------
- [NO]M14 - Release 0.17https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/244Performance improvement on paginated query for CosmosDB2023-10-30T00:36:38ZNeelesh ThakurPerformance improvement on paginated query for CosmosDB## Overview
-------------------------------------
This MR contains an alternative implementation of CosmosStore.queryItemsPage method. Previous implementation have two interactions with a [BlockingIterable](https://github.com/reactor/rea...## Overview
-------------------------------------
This MR contains an alternative implementation of CosmosStore.queryItemsPage method. Previous implementation have two interactions with a [BlockingIterable](https://github.com/reactor/reactor-core/blob/main/reactor-core/src/main/java/reactor/core/publisher/BlockingIterable.java) object that could be reduced to one without lose of meaning. The purpose of such strategy is performance gain once such interactions are time consumers due to its blocking nature.
During the performance tests, we collected the following values
| |Previous Implementation of CosmosStore.queryItemsPage | Latest Implementation of CosmosStore.queryItemsPage |
|--|--|--|
| Batch Size 10000 |TOTAL OPERATION TIME (MS): 249680 | TOTAL OPERATION TIME (MS): 127606 |
| Batch Size 5000 | TOTAL OPERATION TIME (MS): 332617 | TOTAL OPERATION TIME (MS): 180620 |
| Batch Size 2500 | TOTAL OPERATION TIME (MS): 368768 | TOTAL OPERATION TIME (MS): 193582 |
| Batch Size 1000 | TOTAL OPERATION TIME (MS): 448730 | TOTAL OPERATION TIME (MS): 240568 |
This changeset also introduces changes to honor page size for continuation token query and log RUs for that page size.
## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
It was added a new method but no behavior has been changed once the old queryItemsPage was not deleted. It just
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Issue: [28](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/28)
## Does this introduce a breaking change?
-------------------------------------
- [NO]M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/242Update vulnerable dependency: core-common version from 0.16.0 to 0.17.0-rc22023-03-23T19:04:42ZMina OtgonboldUpdate vulnerable dependency: core-common version from 0.16.0 to 0.17.0-rc2This MR updates according to WhiteSource alerts which got updated in core-common:
* [Core-common MR - 167](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/167)
* [Core-common MR - 157](http...This MR updates according to WhiteSource alerts which got updated in core-common:
* [Core-common MR - 167](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/167)
* [Core-common MR - 157](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/157)
## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
Issue: [27](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/27)
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->M14 - Release 0.17https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/240Stack trace Enhancement2023-08-18T12:42:45ZSrishti SharmaStack trace Enhancement## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? Yes
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I ...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? Yes
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes. Yes
* [YES/NO/NA] All new and existing tests passed. Yes
* [YES/NO/NA] My code follows the code style of this project. Yes
* [YES/NO/NA] I ran lint checks locally prior to submission. Yes
## What is the issue or story related to the change?
-------------------------------------
**Issue: **Stack trace is split across multiple line and contains unnecessary information.
**Impact:** Issue is resulting in too many logs. Also impacts DRI's troubleshooting.
**Target: **Reduce stack trace by having each exception without line breaks and removing unwanted details from stacktrace.
High level design:
**PBI:** Product Backlog Item 12836: Enhance exception stack traces - Boards (azure.com)
**Extension of Bug:** Stack traces split across multiple lines - Boards (azure.com)
1. Changes made to log4j2.xml to define the PatternLayout for the exceptions.
2. We filter out packages that would not be necessary for troubleshooting. For example: org.apache,org.springframework,sun.reflect,javax.servlet
3. We remove new lines from the Exception and replace with | separator to avoid exception splitting across multiple lines.
4. Please find below the difference between a segment of generated exception before and after changes: -
**Before:**
2022-09-02 12:36:52.315 ERROR DESKTOP-JF1LE60 --- [nio-8086-exec-6] o.o.o.f.a.FileLocationApi correlation-id=15bcefbe-e482-47b0-a07e-5284580f09d1 data-partition-id=mstest3221-dp api-method=GET operation-name={GET [/v2/files/uploadURL]} user-id=51d2f791-795b-4c8d-9657-cd23b1f9f2a7 app-id=:blah something bad happened java.lang.RuntimeException: koopa
at org.opengroup.osdu.file.api.FileLocationApi.getLocationFile(FileLocationApi.java:81) [file-core-0.17.0-SNAPSHOT.jar!/:0.17.0-SNAPSHOT]
at org.opengroup.osdu.file.api.FileLocationApi$$FastClassBySpringCGLIB$$3bf58739.invoke(<generated>) [file-core-0.17.0-SNAPSHOT.jar!/:0.17.0-SNAPSHOT]
**at org.springframework**.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) [spring-core-5.3.12.jar!/:5.3.12]
**at org.springframework**.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) [spring-aop-5.3.12.jar!/:5.3.12]
**at org.springframework**.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.3.12.jar!/:5.3.12]
**at org.springframework**.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) [spring-aop-5.3.12.jar!/:5.3.12]
**at org.springframework**.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123) [spring-context-5.3.12.jar!/:5.3.12]
**at org.springframework.aop.framework**.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.12.jar!/:5.3.12]
**at org.springframework.aop.framework**.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) [spring-aop-5.3.12.jar!/:5.3.12]
**at org.springframework.aop.framework**.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) [spring-aop-5.3.12.jar!/:5.3.12]
at org.opengroup.osdu.file.api.FileLocationApi$$EnhancerBySpringCGLIB$$ac97402f.getLocationFile(<generated>) [file-core-0.17.0-SNAPSHOT.jar!/:0.17.0-SNAPSHOT]
**After:**
2022-09-05 13:43:34.633 INFO DESKTOP-JF1LE60 --- [nio-8086-exec-4] o.o.o.a.l.Slf4JLogger correlation-id=e1c58641-6788-4a03-bcf0-fd3778a8f2f9 data-partition-id=mstest3221-dp api-method=GET operation-name={GET [/v2/files/uploadURL]} user-id=51d2f791-795b-4c8d-9657-cd23b1f9f2a7 app-id=:file.app method: GET | latency: 0 | url: /groups | correlation id: e1c58641-6788-4a03-bcf0-fd3778a8f2f9 {correlation-id=e1c58641-6788-4a03-bcf0-fd3778a8f2f9, data-partition-id=mstest3221-dp}
2022-09-05 13:43:34.638 ERROR DESKTOP-JF1LE60 --- [nio-8086-exec-4] o.o.o.a.l.Slf4JLogger correlation-id=e1c58641-6788-4a03-bcf0-fd3778a8f2f9 data-partition-id=mstest3221-dp api-method=GET operation-name={GET [/v2/files/uploadURL]} user-id=51d2f791-795b-4c8d-9657-cd23b1f9f2a7 app-id=:file.app null {correlation-id=e1c58641-6788-4a03-bcf0-fd3778a8f2f9, data-partition-id=mstest3221-dp} org.opengroup.osdu.core.common.model.http.AppException: Internal communication failure| at org.opengroup.osdu.core.common.http.HttpClientHandler.sendRequest(HttpClientHandler.java:107) ~[os-core-common-0.16.0.jar!/:?]**| at **org.opengroup.osdu.core.common.http.UrlFetchServiceImpl.sendRequest(UrlFetchServiceImpl.java:64) ~[os-core-common-0.16.0.jar!/:?]| at org.opengroup.osdu.core.common.http.UrlFetchServiceImpl.sendRequest(UrlFetchServiceImpl.java:44) ~[os-core-common-0.16.0.jar!/:?]**| at** org.opengroup.osdu.core.common.http.UrlFetchServiceImpl$$FastClassBySpringCGLIB$$7ffb74e.invoke(<generated>) ~[os-core-common-0.16.0.jar!/:?]| ... **suppressed 9 lines**| at org.opengroup.osdu.core.common.http.UrlFetchServiceImpl$$EnhancerBySpringCGLIB$$e035e205.sendRequest(<generated>) ~[os-core-common-0.16.0.jar!/:?]| at org.opengroup.osdu.azure.httpconfig.HttpClientAzure.decoratedSend(HttpClientAzure.java:63) ~[core-lib-azure-0.17.0-SNAPSHOT.jar!/:?]| at org.opengroup.osdu.azure.httpconfig.HttpClientAzure.send(HttpClientAzure.java:91) ~[core-lib-azure-0.17.0-SNAPSHOT.jar!/:?]| at org.opengroup.osdu.core.common.entitlements.EntitlementsService.getGroups(EntitlementsService.java:77) ~[os-core-common-0.16.0.jar!/:?]| at org.opengroup.osdu.core.common.entitlements.AuthorizationServiceImpl.authorizeAny(AuthorizationServiceImpl.java:52) ~[os-core-common-0.16.0.jar!/:?]| at org.opengroup.osdu.file.middleware.AuthorizationFilter.hasPermission(AuthorizationFilter.java:50) ~[file-core-0.17.0-SNAPSHOT.jar!/:0.17.0-SNAPSHOT]| at org.opengroup.osdu.file.middleware.AuthorizationFilter$$FastClassBySpringCGLIB$$5aded031.invoke(<generated>) ~[file-core-0.17.0-SNAPSHOT.jar!/:0.17.0-SNAPSHOT]| ... suppressed 9 lines| at org.opengroup.osdu.file.middleware.AuthorizationFilter$$EnhancerBySpringCGLIB$$919911a8.hasPermission(<generated>) ~[file-core-0.17.0-SNAPSHOT.jar!/:0.17.0-SNAPSHOT]| ... **suppressed 3 lines**| at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]| ... suppressed 18 lines| at org.opengroup.osdu.file.api.FileLocationApi$$EnhancerBySpringCGLIB$$64470961.getLocationFile(<generated>) ~[file-core-0.17.0-SNAPSHOT.jar!/:0.17.0-SNAPSHOT]| at org.opengroup.osdu.file.api.FileLocationApi$$FastClassBySpringCGLIB$$3bf58739.invoke(<generated>) ~[file-core-0.17.0-SNAPSHOT.jar!/:0.17.0-SNAPSHOT]| ... **suppressed 9 lines**
## Test coverage:
------------------
Tested in IntelliJ Console and Kusto
## Does this introduce a breaking change?
-------------------------------------
- No
## Pending items
----------------
No
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M14 - Release 0.17https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/239Cherry-pick 'Replaced insert bulk operation with upsert for cosmos client' in...2022-12-22T18:58:01ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Replaced insert bulk operation with upsert for cosmos client' into release/0.16Original MR: !237Original MR: !237M13 - Release 0.16David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.org