Storage merge requests
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests
2022-08-12T21:21:46Z
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/420
Add redis properties
2022-08-12T21:21:46Z
Alok Joshi
Add redis properties
Current default redis cache configuration (for groups cache for Azure) sets cache expiration time as 1 hour. This is an issue because 1 hour is too long, any recent changes on user's groups will not be picked up by the cache.
Scenario:
...
Current default redis cache configuration (for groups cache for Azure) sets cache expiration time as 1 hour. This is an issue because 1 hour is too long, any recent changes on user's groups will not be picked up by the cache.
Scenario:
- User makes a Storage call with token 1
- A new group is created with Entitlements service and user is added to that group
- User makes a Storage call which requires user to be in this new group (which he/she is)
With 1 hour cache expiration, storage will fail to sync cache with this new membership, and return 403 for user. Thus, this expiration should be much lower.
M13 - Release 0.16
Alok Joshi
Alok Joshi
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/480
Merge branch 'IBM-webmvc-revert' into 'master'
2022-08-12T20:43:29Z
Shrikant Garg
Merge branch 'IBM-webmvc-revert' into 'master'
Original MR: !479
Original MR: !479
M13 - Release 0.16
David Diederich
d.diederich@opengroup.org
Chad Leong
David Diederich
d.diederich@opengroup.org
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/481
Cherry-pick 'Add redis properties' into release/0.16
2022-08-15T14:45:38Z
David Diederich
d.diederich@opengroup.org
Cherry-pick 'Add redis properties' into release/0.16
Original MR: !420
Original MR: !420
M13 - Release 0.16
David Diederich
d.diederich@opengroup.org
David Diederich
d.diederich@opengroup.org
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/465
Removing Eventgrid Dependency in Storage
2022-08-16T07:02:41Z
Chahat Kumar
Removing Eventgrid Dependency in Storage
# Merge request template
# Merge request template
Chahat Kumar
Chahat Kumar
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/482
Remove SNAPSHOT dependencies
2022-08-17T04:27:55Z
David Diederich
d.diederich@opengroup.org
Remove SNAPSHOT dependencies
This automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### ...
This automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: b654752e9623984448bb6b9f99e5d290e8ba6cd0
Maven: 0.17.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | provider/storage-aws/src/main/ComplianceTrigger/ComplianceTriggerFunction/ComplianceTriggerFunction/ | testing/ |
| ------------------------------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------- | --------------- |
| core-lib-azure | 0.16.0-rc5 | | |
| core-lib-gcp | 0.16.0-rc1 | | |
| os-core-lib-aws | 0.16.0-SNAPSHOT | 0.14.0-rc2 | 0.14.0-rc2 |
| obm | 0.15.0 | | |
| oqm | 0.15.0 | | |
| os-core-common | 0.15.0 | 0.13.0 | 0.13.0 |
| os-core-lib-ibm | 0.16.0-rc1 | | 0.13.0 |
| osm | 0.15.0 | | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.2.2 | 2.6.7.2 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.13.3 | 2.17.1 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.13.3 | 2.17.1 |
| (3rd Party) org.springframework.spring-webflux | 5.3.12 | | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22, 5.1.19.RELEASE | 5.3.12 | 5.3.12 |
```
Critical: Found Vulnerable Spring MVC dependency (<5.2.20 || >=5.3.0 <5.3.18)
├─ _Root_
│ ├─ org.opengroup.osdu.storage-byoc == 0.17.0-SNAPSHOT
│ │ └─ org.opengroup.osdu.storage-core == 0.17.0-SNAPSHOT
│ │ └─ org.springframework.spring-webmvc == 5.1.19.RELEASE
│ └─ org.opengroup.osdu.storage-ibm == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.storage-core == 0.17.0-SNAPSHOT
│ └─ org.springframework.spring-webmvc == 5.1.19.RELEASE
├─ provider/storage-aws/src/main/ComplianceTrigger/ComplianceTriggerFunction/ComplianceTriggerFunction/
│ └─ compliance.compliance == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
└─ testing/
├─ org.opengroup.osdu.storage.storage-test-core == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
├─ org.opengroup.osdu.storage.storage-test-aws == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
├─ org.opengroup.osdu.storage.storage-test-azure == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.storage.storage-test-core == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
├─ org.opengroup.osdu.storage.storage-test-gcp == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.storage.storage-test-core == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
├─ org.opengroup.osdu.storage.storage-test-ibm == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
└─ org.opengroup.osdu.storage.storage-test-anthos == 0.17.0-SNAPSHOT
└─ org.opengroup.osdu.storage.storage-test-core == 0.17.0-SNAPSHOT
└─ org.opengroup.osdu.os-core-common == 0.13.0
└─ org.springframework.boot.spring-boot-starter-web == 2.4.12
└─ org.springframework.spring-webmvc == 5.3.12
```
```
Critical: Found Vulnerable Spring WebFlux dependency (<5.2.20 || >=5.3.0 <5.3.18)
└─ _Root_
└─ org.opengroup.osdu.storage-azure == 0.17.0-SNAPSHOT
└─ com.azure.spring.azure-spring-boot-starter-active-directory == 3.4.0
└─ org.springframework.boot.spring-boot-starter-webflux == 2.4.12
└─ org.springframework.spring-webflux == 5.3.12
```
### Dependency Information After the Upgrade
```
Branch: remove-snapshot-dependencies
SHA: ff7d288f4236b48872b6cdb9d92a50153d98fb61
Maven: 0.17.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | provider/storage-aws/src/main/ComplianceTrigger/ComplianceTriggerFunction/ComplianceTriggerFunction/ | testing/ |
| ------------------------------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------- | --------------- |
| core-lib-azure | 0.16.0-rc5 | | |
| core-lib-gcp | 0.16.0-rc1 | | |
| os-core-lib-aws | 0.16.1 | 0.14.0-rc2 | 0.14.0-rc2 |
| obm | 0.15.0 | | |
| oqm | 0.15.0 | | |
| os-core-common | 0.15.0 | 0.13.0 | 0.13.0 |
| os-core-lib-ibm | 0.16.0-rc1 | | 0.13.0 |
| osm | 0.15.0 | | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.2.2 | 2.6.7.2 | 2.8.1, 2.13.2.2 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.13.3 | 2.17.1 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.13.3 | 2.17.1 |
| (3rd Party) org.springframework.spring-webflux | 5.3.12 | | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.22, 5.1.19.RELEASE | 5.3.12 | 5.3.12 |
```
Critical: Found Vulnerable Spring MVC dependency (<5.2.20 || >=5.3.0 <5.3.18)
├─ _Root_
│ ├─ org.opengroup.osdu.storage-byoc == 0.17.0-SNAPSHOT
│ │ └─ org.opengroup.osdu.storage-core == 0.17.0-SNAPSHOT
│ │ └─ org.springframework.spring-webmvc == 5.1.19.RELEASE
│ └─ org.opengroup.osdu.storage-ibm == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.storage-core == 0.17.0-SNAPSHOT
│ └─ org.springframework.spring-webmvc == 5.1.19.RELEASE
├─ provider/storage-aws/src/main/ComplianceTrigger/ComplianceTriggerFunction/ComplianceTriggerFunction/
│ └─ compliance.compliance == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
└─ testing/
├─ org.opengroup.osdu.storage.storage-test-core == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
├─ org.opengroup.osdu.storage.storage-test-aws == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
├─ org.opengroup.osdu.storage.storage-test-azure == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.storage.storage-test-core == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
├─ org.opengroup.osdu.storage.storage-test-gcp == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.storage.storage-test-core == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
├─ org.opengroup.osdu.storage.storage-test-ibm == 0.17.0-SNAPSHOT
│ └─ org.opengroup.osdu.os-core-common == 0.13.0
│ └─ org.springframework.boot.spring-boot-starter-web == 2.4.12
│ └─ org.springframework.spring-webmvc == 5.3.12
└─ org.opengroup.osdu.storage.storage-test-anthos == 0.17.0-SNAPSHOT
└─ org.opengroup.osdu.storage.storage-test-core == 0.17.0-SNAPSHOT
└─ org.opengroup.osdu.os-core-common == 0.13.0
└─ org.springframework.boot.spring-boot-starter-web == 2.4.12
└─ org.springframework.spring-webmvc == 5.3.12
```
```
Critical: Found Vulnerable Spring WebFlux dependency (<5.2.20 || >=5.3.0 <5.3.18)
└─ _Root_
└─ org.opengroup.osdu.storage-azure == 0.17.0-SNAPSHOT
└─ com.azure.spring.azure-spring-boot-starter-active-directory == 3.4.0
└─ org.springframework.boot.spring-boot-starter-webflux == 2.4.12
└─ org.springframework.spring-webflux == 5.3.12
```
M13 - Release 0.16
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/484
Cherry-pick 'Remove SNAPSHOT dependencies' into release/0.16
2022-08-17T09:12:07Z
David Diederich
d.diederich@opengroup.org
Cherry-pick 'Remove SNAPSHOT dependencies' into release/0.16
Original MR: !482
Original MR: !482
M13 - Release 0.16
David Diederich
d.diederich@opengroup.org
David Diederich
d.diederich@opengroup.org
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/487
Updated entitlements groups (GONRG-3264)
2023-08-18T22:20:53Z
Yurii Ruban [EPAM / GCP]
Updated entitlements groups (GONRG-3264)
# Description:
Updated entitlements groups in readme
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
# Changes in:
- [ ] GCP
# Description:
Updated entitlements groups in readme
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
# Changes in:
- [ ] GCP
M14 - Release 0.17
Rustam Lotsmanenko (EPAM)
rustam_lotsmanenko@epam.com
Rustam Lotsmanenko (EPAM)
rustam_lotsmanenko@epam.com
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/478
Refactor code to SonarQube standards
2022-09-16T02:10:45Z
Morris Estepa
Refactor code to SonarQube standards
M14 - Release 0.17
Morris Estepa
Morris Estepa
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/491
Adding AWS Helm Charts
2022-08-24T12:36:31Z
Marc Burnie [AWS]
Adding AWS Helm Charts
M14 - Release 0.17
Marc Burnie [AWS]
Marc Burnie [AWS]
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/472
GONRG-5117-Refactor variables with a camel case
2023-08-18T22:20:57Z
Yevhenii Orlov (EPAM) [GCP]
GONRG-5117-Refactor variables with a camel case
# Merge request template
# Merge request template
M14 - Release 0.17
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/492
Disable EG after CoreLib Upgrade
2022-12-09T13:41:12Z
Krishna Nikhil Vedurumudi
Disable EG after CoreLib Upgrade
# Merge request template
# Merge request template
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/451
Fix data inconsistency issue between metadata storage and blob storage
2022-09-15T10:04:42Z
Alok Joshi
Fix data inconsistency issue between metadata storage and blob storage
## Issue
We've observed data inconsistency issues for some storage record objects where the latest version of record (available in metadata storage - i.e. cosmosDb) is soft deleted from blob storage (i.e. Azure Storage). This is causing ...
## Issue
We've observed data inconsistency issues for some storage record objects where the latest version of record (available in metadata storage - i.e. cosmosDb) is soft deleted from blob storage (i.e. Azure Storage). This is causing exception for APIs like GET `/api/storage/v2/records/<id>` and POST `/api/storage/v2/query/records:batch`
## Solution
We propose to fix this data inconsistency by two changes:
- Clean up datastore as well, when cleaning up cloud storage, in case of commitBatch method exception in `PersistenceServiceImpl` (core logic change)
- For existing records having this issue: try restoring the data from blob storage. We are attempting to restore the latest record version (recorded in cosmosDb) from blob storage if its not found
M14 - Release 0.17
Alok Joshi
Alok Joshi
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/436
Add CORS header for /record:batch endpoint
2022-09-15T10:07:28Z
Yifan Ye
Add CORS header for /record:batch endpoint
# Merge request template
Fix for issue #132 , add frame-of-reference header to CORS allowed headers
# Merge request template
Fix for issue #132 , add frame-of-reference header to CORS allowed headers
M14 - Release 0.17
Yifan Ye
Yifan Ye
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/490
Change version of azure core lib to 0.17.0-rc7.
2022-09-05T11:48:38Z
Akshat Joshi
Change version of azure core lib to 0.17.0-rc7.
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/496
Update redis expiry
2022-09-06T14:30:25Z
harshit aggarwal
Update redis expiry
# Merge request template
# Merge request template
harshit aggarwal
harshit aggarwal
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/477
[GONRG-5460] Added bootstrap script
2023-08-18T22:20:55Z
Danylo Vanin (EPAM)
[GONRG-5460] Added bootstrap script
# Merge request template
# Merge request template
M14 - Release 0.17
Danylo Vanin (EPAM)
Danylo Vanin (EPAM)
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/495
Updating Core Azure Library for Release/0.16
2022-09-06T20:55:32Z
David Diederich
d.diederich@opengroup.org
Updating Core Azure Library for Release/0.16
This update allows the newly patched content from the Core Azure Library to be used within the Storage service.
The library changes replace bulk operations with upserts; details available at:
osdu/platform/system/lib/cloud/azure/os-...
This update allows the newly patched content from the Core Azure Library to be used within the Storage service.
The library changes replace bulk operations with upserts; details available at:
osdu/platform/system/lib/cloud/azure/os-core-lib-azure!239
This is not a cherry-pick, per se. Because the Core Lib Azure has had several changes for 0.17, the default branch is being updated to incorporate all of them. This MR updates the release branch to use only the 2 patches that were picked into the Core Lib Azure's release branches. Since this exact configuration has never been tested before, I added a trusted branch to the MR to see the results before merging it in.
M13 - Release 0.16
David Diederich
d.diederich@opengroup.org
David Diederich
d.diederich@opengroup.org
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/497
Versioning helm charts
2022-09-06T21:58:27Z
Marc Burnie [AWS]
Versioning helm charts
M14 - Release 0.17
Marc Burnie [AWS]
Okoun-Ola Fabien Houeto
Gustavo Urdaneta
Marc Burnie [AWS]
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/503
Fix OqmSubscriberManager (legaltags_changed topic's subscriptions creation an...
2022-09-13T16:38:45Z
Rustam Lotsmanenko (EPAM)
rustam_lotsmanenko@epam.com
Fix OqmSubscriberManager (legaltags_changed topic's subscriptions creation and usage algorithm) (GONRG-4683)
# Merge request template
# Merge request template
M14 - Release 0.17
Rustam Lotsmanenko (EPAM)
rustam_lotsmanenko@epam.com
Rustam Lotsmanenko (EPAM)
rustam_lotsmanenko@epam.com
https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/500
Stack trace enhancement_Storage Service_master
2022-09-15T09:22:33Z
Srishti Sharma
Stack trace enhancement_Storage Service_master
Issue: Stack trace is split across multiple line and contain unnecessary information. Target: Reduce stack trace by having each exception without line breaks and remove unwanted date. Impact: Too many logs. Difficult to debug for DRIs.
B...
Issue: Stack trace is split across multiple line and contain unnecessary information. Target: Reduce stack trace by having each exception without line breaks and remove unwanted date. Impact: Too many logs. Difficult to debug for DRIs.
Bug: https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/\_workitems/edit/6422 Enhancement: https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/_workitems/edit/12836
1. Updated POM.xml of Storage Service to take up latest code version(0.17.0-rc14)from azure-core-lib i.e. <osdu.corelibazure.version>O.17.0-rc14</osdu.corelibazure.version>
Logging library is already updated with version .17.0-rc14 which removes line breaks in stack trace and filters out unwanted packages' data from stacktrace.
M14 - Release 0.17