OS Core Lib Azure merge requestshttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests2022-11-15T16:30:48Zhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/256Coreservicesupgradev12022-11-15T16:30:48ZNishant VidyasagarCoreservicesupgradev1Adding path matching strategy so, that other dependent services can use this.Adding path matching strategy so, that other dependent services can use this.M15 - Release 0.18Nishant VidyasagarNishant Vidyasagarhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/254coreservices upgrade containing blob storage and file datalake2022-11-15T16:31:19ZNishant Vidyasagarcoreservices upgrade containing blob storage and file datalake## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] All new and existing tests passed.
* [YES] My code follows the code styl...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
For the Azure Blob Storage, we are using the SDK version:- 12.14.2(2021-11-10).The latest SDK version is:- 12.19.1 (2022-09-12) azure-blob-storage SDK version, So, upgraded the same. Since, Intermittent 500s were coming due to azure blob throwing 400, so needed to upgrade the Azure Blob Storage. Upgrading the same added some more dependencies.
## Test coverage:
------------------
ran Service IT's locallyM15 - Release 0.18Nishant VidyasagarNishant Vidyasagarhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/262Update FOSSA NOTICE2022-12-04T08:35:22ZDavid Diederichd.diederich@opengroup.orgUpdate FOSSA NOTICEThis MR updates the attribution file for the project (also known as the `NOTICE` file).
It is important to keep this up to date to satisfy legal requirements of dependency licenses.
We use FOSSA as the tool to scan for and detect these ...This MR updates the attribution file for the project (also known as the `NOTICE` file).
It is important to keep this up to date to satisfy legal requirements of dependency licenses.
We use FOSSA as the tool to scan for and detect these changes.M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/257Expose Collaboration Context in message for new topic (recordsevent topic)2022-12-08T08:03:44ZMina OtgonboldExpose Collaboration Context in message for new topic (recordsevent topic)## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NA] I have updated the documentation accordingly.
* [YES] I have added tests ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NA] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
[ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/149)
Change details:
The message for the new topic (recordsevent) contains collaboration context if provided
## Test coverage:
------------------
Added unit test to verify that the new message contains collaboration context if provided
## Does this introduce a breaking change?
-------------------------------------
- [NO]M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/263Upgrade First Party Library Dependencies for Release 0.182022-12-08T08:36:39ZDavid Diederichd.diederich@opengroup.orgUpgrade First Party Library Dependencies for Release 0.18This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any...This automated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep the OSDU projects utilizing the latest available code to ensure widespread usage and stability.
However, any library that is older than the previous release will be left as-is, since the upgrade is likely to be more complicated.
Furthermore, the upgrade should only be merged in the CI pipeline reports success.
If this MR has failed, we can spend a little time investigating to see if a trivial upgrade could achieve compatiblity to the new library.
But significant upgrade efforts should not occur on this MR, as part of the release tagging process.
Instead, significant work should be scheduled for a subsequent milestone.
### Dependency Information Before the Upgrade
```
Branch: release/0.18
SHA: eb79ab42caf333e3acc0756f6c5803f474ff21df
Maven: 0.18.0-SNAPSHOT
```
| Maven Dependencies | _Root_ |
| ------------------------------------------------------- | ---------- |
| os-core-common | 0.18.0-rc3 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.4 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: c3d208a06ab1865292173916fa9decbe7b83e0c0
Maven: 0.19.0-SNAPSHOT
```
| Maven Dependencies | _Root_ |
| ------------------------------------------------------- | ------ |
| os-core-common | 0.18.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.4 |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22 |
# Why is this a direct patch?
The default branch has already moved on to os-core-common `0.19.0-rc1`.
So, there's no benefit in merging this to the default branch (and it would undo work already started towards the next release), but we need it here for the 0.18 release.M15 - Release 0.18David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/264Cherry-pick 'Expose Collaboration Context in message for new topic (recordsev...2022-12-08T17:28:59ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Expose Collaboration Context in message for new topic (recordsevent topic)' into release/0.18**Original MR**: !257
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !257
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/pipelines/new?ref=cherry-pick-for-257)M15 - Release 0.18David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/268Upgrade com.azure:azure-core to latest version 1.34.02022-12-15T17:57:07ZThulasi Dass SubramanianUpgrade com.azure:azure-core to latest version 1.34.0- Upgrade com.azure:azure-core to latest version 1.34.0- Upgrade com.azure:azure-core to latest version 1.34.0M15 - Release 0.18Thulasi Dass SubramanianThulasi Dass Subramanianhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/270Cherry-pick 'Upgrade com.azure:azure-core to latest version 1.34.0' into rele...2022-12-15T17:58:21ZChad LeongCherry-pick 'Upgrade com.azure:azure-core to latest version 1.34.0' into release/0.18**Original MR**: !268
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !268
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/pipelines/new?ref=cherry-pick-for-268)M15 - Release 0.18David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/259Merge branch 'ankurrawat/ErrorMessageFix' into 'release/0.17'2022-12-22T15:08:57ZAnkur RawatMerge branch 'ankurrawat/ErrorMessageFix' into 'release/0.17'4xx error handling for bulkUpdate Operation4xx error handling for bulkUpdate OperationM15 - Release 0.18Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/255Added a regex filter to filter out a particular pattern of msgs received from...2023-08-18T12:42:35ZNaga Aneesh MylavarapuAdded a regex filter to filter out a particular pattern of msgs received from core message receiver## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_sprints/taskboard/Energy%20-%20Explorers/Open%20Energy%20Platform/Oct%20'22/Oct%20'22%20Sprint%203?workitem=7737
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details: Added a regex filter to filter out messages of a particular pattern received from CoreMessageReceiver. Added a threshold filter to ensure that messages are removed only if their level is info or below.
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO] No
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M15 - Release 0.18Naga Aneesh MylavarapuNaga Aneesh Mylavarapuhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/250Update src/main/java/org/opengroup/osdu/azure/publisherFacade/PubsubConfigura...2023-08-18T12:42:37ZNikhil Singh[MicroSoft]Update src/main/java/org/opengroup/osdu/azure/publisherFacade/PubsubConfiguration.java## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
* [YES/NO/NA] My code follows the code style of this project.
* [YES/NO/NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/249increase version of os-core-common to 0.18.0-rc32023-08-18T12:42:38ZYauheni Lesnikauincrease version of os-core-common to 0.18.0-rc3increase version of os-core-common to 0.18.0-rc3 to fix CVE-2022-42004 whitesource security vulnerability for the jackson-databind library
Related MR: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_...increase version of os-core-common to 0.18.0-rc3 to fix CVE-2022-42004 whitesource security vulnerability for the jackson-databind library
Related MR: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/182M15 - Release 0.18Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/248increase version of os-core-common to 0.18.0-rc22023-08-18T12:42:40ZYauheni Lesnikauincrease version of os-core-common to 0.18.0-rc2Increase version of os-core-common with to 0.18.0-rc2 fix of whitesource security vulnerabilities.
Merged MR of os-core-common: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/179
Fix white...Increase version of os-core-common with to 0.18.0-rc2 fix of whitesource security vulnerabilities.
Merged MR of os-core-common: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/179
Fix whitesource high severity security vulnerabilities for next dependencies:
snakeyaml-1.26.jar - CVE-2022-25857
jackson-databind-2.13.2 - CVE-2022-42004M15 - Release 0.18Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/244Performance improvement on paginated query for CosmosDB2023-10-30T00:36:38ZNeelesh ThakurPerformance improvement on paginated query for CosmosDB## Overview
-------------------------------------
This MR contains an alternative implementation of CosmosStore.queryItemsPage method. Previous implementation have two interactions with a [BlockingIterable](https://github.com/reactor/rea...## Overview
-------------------------------------
This MR contains an alternative implementation of CosmosStore.queryItemsPage method. Previous implementation have two interactions with a [BlockingIterable](https://github.com/reactor/reactor-core/blob/main/reactor-core/src/main/java/reactor/core/publisher/BlockingIterable.java) object that could be reduced to one without lose of meaning. The purpose of such strategy is performance gain once such interactions are time consumers due to its blocking nature.
During the performance tests, we collected the following values
| |Previous Implementation of CosmosStore.queryItemsPage | Latest Implementation of CosmosStore.queryItemsPage |
|--|--|--|
| Batch Size 10000 |TOTAL OPERATION TIME (MS): 249680 | TOTAL OPERATION TIME (MS): 127606 |
| Batch Size 5000 | TOTAL OPERATION TIME (MS): 332617 | TOTAL OPERATION TIME (MS): 180620 |
| Batch Size 2500 | TOTAL OPERATION TIME (MS): 368768 | TOTAL OPERATION TIME (MS): 193582 |
| Batch Size 1000 | TOTAL OPERATION TIME (MS): 448730 | TOTAL OPERATION TIME (MS): 240568 |
This changeset also introduces changes to honor page size for continuation token query and log RUs for that page size.
## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
It was added a new method but no behavior has been changed once the old queryItemsPage was not deleted. It just
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
High level design:
Issue: [28](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/28)
## Does this introduce a breaking change?
-------------------------------------
- [NO]M15 - Release 0.18