Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2022-12-09T19:39:22Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/575Cherry-pick 'Update Storage to be Collaboration Context Aware' into release/0.182022-12-09T19:39:22ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Update Storage to be Collaboration Context Aware' into release/0.18**Original MR**: !546
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !546
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-546)M15 - Release 0.18David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/574GONRG-5937: Merge GC Helms to the common one2023-01-12T23:53:23ZVolodymyr Pienskoi [EPAM / GCP]GONRG-5937: Merge GC Helms to the common oneM16 - Release 0.19Volodymyr Pienskoi [EPAM / GCP]Volodymyr Pienskoi [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/573Fixed vulnerabilities2022-12-13T12:07:51ZVineeth Guna [Microsoft]Fixed vulnerabilities# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/572Cherry-pick 'ADR 92: Populate recordBlocks in record modification events' int...2022-12-08T17:31:56ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'ADR 92: Populate recordBlocks in record modification events' into release/0.18**Original MR**: !539
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !539
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-539)M15 - Release 0.18David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/567Update FOSSA NOTICE2022-12-05T22:21:36ZDavid Diederichd.diederich@opengroup.orgUpdate FOSSA NOTICEThis MR updates the attribution file for the project (also known as the `NOTICE` file).
It is important to keep this up to date to satisfy legal requirements of dependency licenses.
We use FOSSA as the tool to scan for and detect these ...This MR updates the attribution file for the project (also known as the `NOTICE` file).
It is important to keep this up to date to satisfy legal requirements of dependency licenses.
We use FOSSA as the tool to scan for and detect these changes.M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/566Aws integration merge2022-12-02T22:15:11ZMichael SaccoAws integration mergeUpdate com.google.oauth-client library to version 1.34.1Update com.google.oauth-client library to version 1.34.1M15 - Release 0.18Michael SaccoMichael Saccohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/560added spring-boot-maven plugin version2022-11-30T06:40:51ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comadded spring-boot-maven plugin version## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] Google Cloud
- [x] IBM
- [x] Common code
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Projects that do not specify the spring-boot-maven plugin version will not compile, since the latest version of the spring-boot-maven plugin suppose to build spring-boot V3 projects with Java 17M15 - Release 0.18Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/559Storage service version upgradation after corelib azure upgrade2022-12-02T05:33:54ZNishant VidyasagarStorage service version upgradation after corelib azure upgradeNishant VidyasagarNishant Vidyasagarhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/556added changes to fix the azure_code_coverage job2022-12-01T09:24:14Zshivani karipeadded changes to fix the azure_code_coverage job- This change will fix the azure code coverage job and produce the code coverage results
- Jacoco plugin is used for the code coverage
- issue ref: https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-prov...- This change will fix the azure code coverage job and produce the code coverage results
- Jacoco plugin is used for the code coverage
- issue ref: https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/244M15 - Release 0.18shivani karipeshivani karipehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/554Revert "Merge branch 'gcp-password-for-redis' into 'master'"2023-08-18T22:20:45ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRevert "Merge branch 'gcp-password-for-redis' into 'master'"# Merge request template# Merge request templateM15 - Release 0.18Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/553Include collaboration context when pushing to new record changed topic2024-03-19T02:18:18ZMina OtgonboldInclude collaboration context when pushing to new record changed topic## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IB...## Type of change
- [ ] Bug Fix
- [x] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
If collaboration context is provided in the request: Azure message bus neither sends a message to recordstopic (i.e. let's call it an old topic) nor recordsevent (i.e. let's call it a new topic)
If collaboration context is not provided in the request: Azure message bus sends messages to the old topic (recordstopic) only
## What is the new/expected behavior?
If collaboration context is provided in the request: Azure message bus sends message to a new topic (recordsevent) that contains x-collaboration directives in message properties.
If collaboration context is not provided in the request: Azure message bus sends message to new topic (recordsevent) and old topic (recordstopic)
## Have you added/updated Unit Tests and Integration Tests?
Yes
## Any other useful information
[ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/149)
This [core-lib-azure MR](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/257) has to merge to master before this MR.M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/552upgrading vulnerability dependency2022-11-24T05:29:03ZAshwani Pandeyupgrading vulnerability dependencyUpgraded dependencies to fix below vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21875
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21...Upgraded dependencies to fix below vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21875
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21868
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21867
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18898
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18896
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18894
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18892
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18886
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18885
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18883
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18880
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18879
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18875
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18874
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24810
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24803
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24798
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24797
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24795
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24794
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24253
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24251
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24250
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24249
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24246
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21874
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21860
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21857
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21851
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21294
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21293
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/20014
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/20013
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/20011
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/20006
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/19440
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/19436
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/16483
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/16482
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/12481M15 - Release 0.18Ashwani PandeyAshwani Pandeyhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/551GONRG-6036: Add vars for opa2023-08-18T22:20:47ZYauheni Rykhter (EPAM)GONRG-6036: Add vars for opa# Merge request template# Merge request templateM15 - Release 0.18Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/550Removing the azure message bus message id2022-12-14T05:51:04ZThiago SenadorRemoving the azure message bus message id## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IB...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Azure message bus message id is composed by correlation id. We have applications that use the same correlation in multiple requests to storage and that is creating duplicated messages with incorrect behavior.
## What is the new/expected behavior?
Azure message bus message id is going to be assigned by Azure itself, guaranteeing its uniqueness.
## Have you added/updated Unit Tests and Integration Tests?
No
## Any other useful informationM15 - Release 0.18https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/548M14 Upgrade2022-12-06T09:32:00ZAnkur RawatM14 UpgradeM12 changes into M14M12 changes into M14Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/546Update Storage to be Collaboration Context Aware2022-12-09T15:57:24ZMina OtgonboldUpdate Storage to be Collaboration Context Aware# Merge request template
This MR updates all APIs in storage to be collaboration context-aware. This functionality is behind a collaboration feature flag which is set to false by default. The functionality of the existing storage service...# Merge request template
This MR updates all APIs in storage to be collaboration context-aware. This functionality is behind a collaboration feature flag which is set to false by default. The functionality of the existing storage service will not be changed with this feature flag set to false.
When it is set to true the old functionality is still not changed however you can work with Records in new contexts using the x-collaboration header when it is optionally provided.
[This core-common MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/183) has to be merged prior to this PR.
This aligns with the ADR below. This MR does not include the new Record changed topic. The current record changed topic is not triggered by any changes when a collaboration context is provided.
References
- [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/149)M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/545Squashed commit of the following2022-11-21T16:40:42ZMorris EstepaSquashed commit of the followingupdate buildspec & pom versionsupdate buildspec & pom versionsM15 - Release 0.18Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/543fix null pointer exception2022-12-27T05:33:19ZNikhil Singh[MicroSoft]fix null pointer exception**Introduction:**
Storage service is giving 500s due to null pointer exception when trying to convert ACL viewers into an array list but we dont have any such log for same. Also it is not reproducible because we never ingest a record wi...**Introduction:**
Storage service is giving 500s due to null pointer exception when trying to convert ACL viewers into an array list but we dont have any such log for same. Also it is not reproducible because we never ingest a record with acl fields empty. Hence we want to log the statement if ACL is empty and throw a NPE with proper reasoning.
**Exception:**
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)| at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)| at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)| at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)| at java.lang.Thread.run(Thread.java:750)|Caused by: java.lang.NullPointerException| at org.opengroup.osdu.storage.provider.azure.CloudStorageImpl.**hasViewerAccessToRecord(CloudStorageImpl.java:238)**| at org.opengroup.osdu.storage.provider.azure.CloudStorageImpl.read(CloudStorageImpl.java:284)| at org.opengroup.osdu.storage.provider.azure.CloudStorageImpl$$FastClassBySpringCGLIB$$47bbc639.invoke(<generated>)| at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)| at
ref #151Nikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/542Changed CRC32C to MD5 to calculate a checksum (GONRG-5884)2022-12-12T23:54:41ZRiabokon Stanislav(EPAM)[GCP]Changed CRC32C to MD5 to calculate a checksum (GONRG-5884)## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IB...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the new/expected behavior?
Upgraded obm lib. look at https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/merge_requests/57.
Changed an algorithm to generate check sum.M15 - Release 0.18Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/540added ability to config redis password and ssl (GONRG-5301)2022-12-13T00:28:53ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comadded ability to config redis password and ssl (GONRG-5301)added ability to config redis password and ssladded ability to config redis password and sslM15 - Release 0.18Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]