Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2024-03-22T15:29:36Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/855Spring 6 Migration - Code changes2024-03-22T15:29:36ZGuillaume CailletSpring 6 Migration - Code changesJavax to Jakarta migration isolated into a dedicated MR: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/854Javax to Jakarta migration isolated into a dedicated MR: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/854M23 - Release 0.26Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/854Spring 6 Migration2024-03-25T15:29:33ZGuillaume CailletSpring 6 Migration# Merge request template# Merge request templateM23 - Release 0.26Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/853Draft: Gonrg 9288 update core plus pipeline2024-03-29T09:06:04ZSiarhei Poliak [EPAM / GCP]Draft: Gonrg 9288 update core plus pipeline# Merge request template# Merge request templateM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/852Aws upgrade aws lib2024-03-08T16:15:47ZDerek HudsonAws upgrade aws lib# Merge request template# Merge request templateM23 - Release 0.26Derek HudsonDerek Hudsonhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/851[#MSCOSDU-35822] upgrade : core-lib-azure Java 17 version2024-03-08T04:25:23ZThulasi Dass Subramanian[#MSCOSDU-35822] upgrade : core-lib-azure Java 17 version## Change details
* For Azure only: upgraded `core-lib-azure` version to `0.26.0-rc5` for Java 17 update in source modules
## Changes in:
* [ ] GCP
* [x] Azure
* [ ] AWS
* [ ] IBM## Change details
* For Azure only: upgraded `core-lib-azure` version to `0.26.0-rc5` for Java 17 update in source modules
## Changes in:
* [ ] GCP
* [x] Azure
* [ ] AWS
* [ ] IBMM23 - Release 0.26Thulasi Dass SubramanianThulasi Dass Subramanianhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/850[GONRG-9211] Added labels to gc helm2024-03-19T17:13:53ZDanylo Vanin (EPAM)[GONRG-9211] Added labels to gc helmAdded labels to GC helmAdded labels to GC helmM23 - Release 0.26Danylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/849Aws code quality improvement2024-02-29T16:17:01ZBruce JinAws code quality improvementThis MR will improve AWS code qualityThis MR will improve AWS code qualityM23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/848[MSCOSDU-3593]Adding core code coverage2024-03-04T05:51:23ZDeepa Kumari[MSCOSDU-3593]Adding core code coverage1. Add code coverage for storage-core module
2. Moved the tests to Junit-5 from Junit-41. Add code coverage for storage-core module
2. Moved the tests to Junit-5 from Junit-4M23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/847Replay feature 2.02024-03-26T19:12:00ZAkshat JoshiReplay feature 2.0## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- Yes
## Does this introduce a change in the cloud provid...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- Yes
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- No
## What is the current behavior?
-Not applicable as the feature introduce new api.
## What is the new/expected behavior?
-Not applicable as the feature introduce new api.
## Have you added/updated Unit Tests and Integration Tests?
-Yes
## Any other useful information
These are the list of ADR related to this feature -
- [ADR Replay](https://community.opengroup.org/osdu/platform/system/storage/-/issues/186)
- [ADR Replay API](https://community.opengroup.org/osdu/platform/system/storage/-/issues/187)
- [Replay Consumer Topic Identification](https://community.opengroup.org/osdu/platform/system/storage/-/issues/190)M23 - Release 0.26Akshat JoshiAkshat Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/846Replay Feature2024-02-25T11:40:30ZAkshat JoshiReplay Feature# Merge request template# Merge request templatehttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/845Update os-core-lib-aws2024-03-19T17:13:54ZRucha DeshpandeUpdate os-core-lib-aws# Merge request template# Merge request templateM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/843auth test cases fix2024-03-19T17:13:56ZNikhil Patilnikhil.patil5@ibm.comauth test cases fix# Merge request template# Merge request templateM23 - Release 0.26Nikhil Patilnikhil.patil5@ibm.comNikhil Patilnikhil.patil5@ibm.comhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/842minor fix on aws region2024-02-06T16:24:38ZYunhua Koglinminor fix on aws region# Merge request template# Merge request templateM23 - Release 0.26Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/839allow id with dot if request with and without dots are in separate requests2024-02-01T22:44:08ZNeelesh Thakurallow id with dot if request with and without dots are in separate requestsEarlier fix on the issue #213 was a breaking change for the consumers. This change-set mitigate the breaking change to certain extent.
OSDU record-ids support trailing dot (.) as defined by Data Definition [schema pattern](https://gitla...Earlier fix on the issue #213 was a breaking change for the consumers. This change-set mitigate the breaking change to certain extent.
OSDU record-ids support trailing dot (.) as defined by Data Definition [schema pattern](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blame/master/Generated/abstract/AbstractSystemProperties.1.0.0.json?ref_type=heads#L14). This exact pattern is also used by Storage service's [record-id validator](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blame/1ce74b5eaebdf6b4390c1b50d3ad0e4b76b46e73/src/main/java/org/opengroup/osdu/core/common/model/storage/validation/ValidationDoc.java#L25). As this pattern is defined on reference schema, and used by almost all schemas on OSDU Data Platform, any change to record-id pattern will require major version upgrade to virtually all schemas.
As mentioned on the issue, [Azure Cloud Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata) does NOT support trailing dot (.) and hence similar shortcoming are imposed on Storage API.
There are few solutions that can be implemented to address shortcoming in Azure Cloud Storage, but none of those solutions are trivial and require lot of refactoring of common code or low level cloud storage classes on Storage service.
The specific issue only happens when record with and without trailing dot (.) are used on same PUT API call. If they are split in different requests then we can process such record-ids. Moreover, usage of such record-ids are only observed in reference-schema's OSDU bootstrapped data and it's not widely used.
This changeset loosen up previous [validation ](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/825) that outright reject all requests with trailing dot (.). It will still be a breaking change if users have automated workflow to create records that includes record-ids with and without trailing dot (.) on same request, but it will not completely drop the support the record-id with trailing dot (.).M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/837Revert a change that causes long messageIds that can lead to failure in publi...2024-02-06T11:27:29ZSabarish K R ERevert a change that causes long messageIds that can lead to failure in publishing
Reverts PR https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/751
As part of the change being reverted, The UUID and the correlationID were being used as messaageID, to enable traceability. The change was cau...
Reverts PR https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/751
As part of the change being reverted, The UUID and the correlationID were being used as messaageID, to enable traceability. The change was causing failures when the input correlation ID was longer than 91 characters, as it was being appended with a random UUID that was 36 characters. Using string with > 128 characters as the messageId when publishing a message to ServiceBus leads to the publish operation failing.
We need to revert this change for now and tackle the traceability in a different way in future.Abhishek PatilJANRAJ CJAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/836fix: fix in Core to ensure inappropriate headers removed while sending reques...2024-02-20T12:53:10ZAnuj Guptafix: fix in Core to ensure inappropriate headers removed while sending request...# Merge request template
https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/670
Reference to old MR introducing the issue: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/252 and https://c...# Merge request template
https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/670
Reference to old MR introducing the issue: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/252 and https://community.opengroup.org/osdu/platform/system/storage/-/issues/55M23 - Release 0.26Ashwani Pandeyvikas ranaAshwani Pandeyhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/835Resolve AWS internal scanned CVE2024-01-26T14:11:45ZBruce JinResolve AWS internal scanned CVEResolve AWS internal trivy scan CVE, will need to skip `logback` related CVEs since these need to be solved after Spring 6.0 upgradeResolve AWS internal trivy scan CVE, will need to skip `logback` related CVEs since these need to be solved after Spring 6.0 upgradeM23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/834[MSCOSDU-2064] Added gcs version paths validation for update record flow2024-01-29T04:40:27ZDeepa Kumari[MSCOSDU-2064] Added gcs version paths validation for update record flowAs per #178, the GCS version array size grows huge causing issues. It was required to not allow update of a record where gcs array size is greater than a configurable specific soft limit. This change is sitting behind a feature flag.
Th...As per #178, the GCS version array size grows huge causing issues. It was required to not allow update of a record where gcs array size is greater than a configurable specific soft limit. This change is sitting behind a feature flag.
The implementation will ignore the records with greater versions than the soft limit and a warning will be logged in application logs.M23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/833enable collaboration header2024-03-19T17:13:58ZYunhua Koglinenable collaboration header# Merge request template# Merge request templateM23 - Release 0.26Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/829Fix vulnerbilities2024-01-23T14:38:11ZBruce JinFix vulnerbilitiesFix existing critical CVEs and other Vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/issues/196
https://community.opengroup.org/osdu/platform/system/storage/-/issues/194
https://community.opengroup.org/osd...Fix existing critical CVEs and other Vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/issues/196
https://community.opengroup.org/osdu/platform/system/storage/-/issues/194
https://community.opengroup.org/osdu/platform/system/storage/-/issues/197
https://community.opengroup.org/osdu/platform/system/storage/-/issues/198
https://community.opengroup.org/osdu/platform/system/storage/-/issues/199
https://community.opengroup.org/osdu/platform/system/storage/-/issues/200
https://community.opengroup.org/osdu/platform/system/storage/-/issues/201
https://community.opengroup.org/osdu/platform/system/storage/-/issues/202
https://community.opengroup.org/osdu/platform/system/storage/-/issues/203M23 - Release 0.26Bruce JinBruce Jin