Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2022-12-06T09:32:00Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/548M14 Upgrade2022-12-06T09:32:00ZAnkur RawatM14 UpgradeM12 changes into M14M12 changes into M14Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/547m14 upgrade2022-11-16T13:00:43ZAnkur Rawatm14 upgradeM12 changes into M14M12 changes into M14Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/546Update Storage to be Collaboration Context Aware2022-12-09T15:57:24ZMina OtgonboldUpdate Storage to be Collaboration Context Aware# Merge request template
This MR updates all APIs in storage to be collaboration context-aware. This functionality is behind a collaboration feature flag which is set to false by default. The functionality of the existing storage service...# Merge request template
This MR updates all APIs in storage to be collaboration context-aware. This functionality is behind a collaboration feature flag which is set to false by default. The functionality of the existing storage service will not be changed with this feature flag set to false.
When it is set to true the old functionality is still not changed however you can work with Records in new contexts using the x-collaboration header when it is optionally provided.
[This core-common MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/183) has to be merged prior to this PR.
This aligns with the ADR below. This MR does not include the new Record changed topic. The current record changed topic is not triggered by any changes when a collaboration context is provided.
References
- [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/149)M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/545Squashed commit of the following2022-11-21T16:40:42ZMorris EstepaSquashed commit of the followingupdate buildspec & pom versionsupdate buildspec & pom versionsM15 - Release 0.18Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/544upgrading vulnerable dependencies2022-11-22T11:54:45ZAshwani Pandeyupgrading vulnerable dependenciesUpgraded dependencies to fix below vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21875
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21...Upgraded dependencies to fix below vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21875
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21868
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21867
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18898
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18896
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18894
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18892
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18886
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18885
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18883
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18880
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18879
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18875
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/18874
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24810
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24803
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24798
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24797
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24795
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24794
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24253
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24251
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24250
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24249
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/24246
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21874
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21860
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21857
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21851
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21294
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/21293
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/20014
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/20013
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/20011
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/20006
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/19440
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/19436
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/16483
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/16482
https://community.opengroup.org/osdu/platform/system/storage/-/security/vulnerabilities/12481M15 - Release 0.18Ashwani PandeyAshwani Pandeyhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/543fix null pointer exception2022-12-27T05:33:19ZNikhil Singh[MicroSoft]fix null pointer exception**Introduction:**
Storage service is giving 500s due to null pointer exception when trying to convert ACL viewers into an array list but we dont have any such log for same. Also it is not reproducible because we never ingest a record wi...**Introduction:**
Storage service is giving 500s due to null pointer exception when trying to convert ACL viewers into an array list but we dont have any such log for same. Also it is not reproducible because we never ingest a record with acl fields empty. Hence we want to log the statement if ACL is empty and throw a NPE with proper reasoning.
**Exception:**
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)| at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)| at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)| at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)| at java.lang.Thread.run(Thread.java:750)|Caused by: java.lang.NullPointerException| at org.opengroup.osdu.storage.provider.azure.CloudStorageImpl.**hasViewerAccessToRecord(CloudStorageImpl.java:238)**| at org.opengroup.osdu.storage.provider.azure.CloudStorageImpl.read(CloudStorageImpl.java:284)| at org.opengroup.osdu.storage.provider.azure.CloudStorageImpl$$FastClassBySpringCGLIB$$47bbc639.invoke(<generated>)| at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)| at
ref #151Nikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/542Changed CRC32C to MD5 to calculate a checksum (GONRG-5884)2022-12-12T23:54:41ZRiabokon Stanislav(EPAM)[GCP]Changed CRC32C to MD5 to calculate a checksum (GONRG-5884)## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IB...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the new/expected behavior?
Upgraded obm lib. look at https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/merge_requests/57.
Changed an algorithm to generate check sum.M15 - Release 0.18Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/541NPE handling2023-08-25T22:28:45ZNikhil Singh[MicroSoft]NPE handling# Merge request template# Merge request templateNikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/539ADR 92: Populate recordBlocks in record modification events2022-12-08T22:56:14ZAbhishek NandaADR 92: Populate recordBlocks in record modification events## Type of change
- [x] Bug Fix
- [X] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[ADR 92](https://community.opengroup.org/osdu/platform/system/home/-/issues/92)
## Does this introduce a chang...## Type of change
- [x] Bug Fix
- [X] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[ADR 92](https://community.opengroup.org/osdu/platform/system/home/-/issues/92)
## Does this introduce a change in the core logic?
- [YES/NO] No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [x] IBM (Added an additional field (hash) to the document db. Auto populated for all other providers.)
## Does this introduce a breaking change?
- [YES/NO] No
## What is the current behavior?
- skipdupes only works for changes in RecordData and ignores changes in RecordMetadata.
- Record modification events do not contain info on which section was updated
## What is the new/expected behavior?
- skipdupes=true now handles for change in any of RecordData and RecordMetadata fields
- This also implements [ADR 92](https://community.opengroup.org/osdu/platform/system/home/-/issues/92) which populates a recordBlocks field with the fields updated
- skipdupes now uses the recordBlock field to check if any updates are made to the record
## Have you added/updated Unit Tests and Integration Tests?
Unit Tests for all new files and new lines with 100% coverage has been added
## Any other useful information
The hashes of data and meta field are now saved in metadata which is used to validate if any changes were made and prevents and extra call to the blob storage for each record.
This is used both in skipdupes=true scenario as well as to populate the recordBlocks field.
This brings around a 50% performance improvement in skipdupes=true scenariosM15 - Release 0.18https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/538GONRG-5890-Change-to-GC2023-08-18T22:20:48ZEldor Pulatov [EPAM/GCP]GONRG-5890-Change-to-GC# Merge request template# Merge request templateM15 - Release 0.18Eldor Pulatov [EPAM/GCP]Eldor Pulatov [EPAM/GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/537Upgrade spring-security-core2022-10-26T18:34:10ZXiangliang MengUpgrade spring-security-coreOkoun-Ola Fabien HouetoXiangliang MengOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/536DO NOT MERGE2022-12-22T18:20:13Zshivani karipeDO NOT MERGE# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/535Storage Performance (GONRG-4754)2022-12-13T00:35:58ZRiabokon Stanislav(EPAM)[GCP]Storage Performance (GONRG-4754)## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] I...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the new/expected behavior?
Fixed NPE. Key is Null.
```
{"code":500,"reason":"Server error.","message":"An unknown error has occurred."}
{"code":500,"reason":"Error writing record.","message":"The server could not process your request at the moment."}
```
https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/osm/-/merge_requests/31/M15 - Release 0.18Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/534Delete all versions of the blob when purging a record2022-12-09T10:53:01ZAlok JoshiDelete all versions of the blob when purging a recordPlease refer to [issue ](https://community.opengroup.org/osdu/platform/system/storage/-/issues/147)
This change solves the samePlease refer to [issue ](https://community.opengroup.org/osdu/platform/system/storage/-/issues/147)
This change solves the sameM15 - Release 0.18Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/533Upgrade spring-beans2023-08-18T22:20:50ZXiangliang MengUpgrade spring-beans# Merge request template# Merge request templateM15 - Release 0.18Okoun-Ola Fabien HouetoXiangliang MengOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/532consume latest version of core common library and update the way of getting p...2022-12-12T22:51:21ZKelly Zhouconsume latest version of core common library and update the way of getting parent recordsPlease refer to issue [here](https://community.opengroup.org/osdu/platform/system/storage/-/issues/145)Please refer to issue [here](https://community.opengroup.org/osdu/platform/system/storage/-/issues/145)M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/531Correct the attribute name to "PreloadFilePath" for os-storage2023-08-18T22:20:52ZXiangliang MengCorrect the attribute name to "PreloadFilePath" for os-storage# Merge request template# Merge request templateM15 - Release 0.18Okoun-Ola Fabien HouetoXiangliang MengOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/530Whitesource update vulnerable dependencies2023-05-10T17:48:30ZMykyta SavchukWhitesource update vulnerable dependenciesUpdated common libraries, but because spring boot was updated to 2.7.2 there, so I added some code changes:
- Migrated from springfox to springdoc, because springfox is not compatible with this version of spring boot
- The new version of...Updated common libraries, but because spring boot was updated to 2.7.2 there, so I added some code changes:
- Migrated from springfox to springdoc, because springfox is not compatible with this version of spring boot
- The new version of spring boot found dependency cycle, so I removed it with @LazyM15 - Release 0.18Mykyta SavchukMykyta Savchukhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/529Cherry-pick 'Merge ibm helm' into release/0.172022-10-10T18:09:39ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Merge ibm helm' into release/0.17**Original MR**: !526
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !526
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-526)M14 - Release 0.17David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/528Cherry-pick 'Added valid access token for test case inputs' into release/0.172022-10-07T07:51:45ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Added valid access token for test case inputs' into release/0.17**Original MR**: !508
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !508
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/storage/-/pipelines/new?ref=cherry-pick-for-508)M14 - Release 0.17David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.org