Storage merge requestshttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests2024-02-06T16:24:38Zhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/842minor fix on aws region2024-02-06T16:24:38ZYunhua Koglinminor fix on aws region# Merge request template# Merge request templateM23 - Release 0.26Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/837Revert a change that causes long messageIds that can lead to failure in publi...2024-02-06T11:27:29ZSabarish K R ERevert a change that causes long messageIds that can lead to failure in publishing
Reverts PR https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/751
As part of the change being reverted, The UUID and the correlationID were being used as messaageID, to enable traceability. The change was cau...
Reverts PR https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/751
As part of the change being reverted, The UUID and the correlationID were being used as messaageID, to enable traceability. The change was causing failures when the input correlation ID was longer than 91 characters, as it was being appended with a random UUID that was 36 characters. Using string with > 128 characters as the messageId when publishing a message to ServiceBus leads to the publish operation failing.
We need to revert this change for now and tackle the traceability in a different way in future.Abhishek PatilJANRAJ CJAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/839allow id with dot if request with and without dots are in separate requests2024-02-01T22:44:08ZNeelesh Thakurallow id with dot if request with and without dots are in separate requestsEarlier fix on the issue #213 was a breaking change for the consumers. This change-set mitigate the breaking change to certain extent.
OSDU record-ids support trailing dot (.) as defined by Data Definition [schema pattern](https://gitla...Earlier fix on the issue #213 was a breaking change for the consumers. This change-set mitigate the breaking change to certain extent.
OSDU record-ids support trailing dot (.) as defined by Data Definition [schema pattern](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blame/master/Generated/abstract/AbstractSystemProperties.1.0.0.json?ref_type=heads#L14). This exact pattern is also used by Storage service's [record-id validator](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blame/1ce74b5eaebdf6b4390c1b50d3ad0e4b76b46e73/src/main/java/org/opengroup/osdu/core/common/model/storage/validation/ValidationDoc.java#L25). As this pattern is defined on reference schema, and used by almost all schemas on OSDU Data Platform, any change to record-id pattern will require major version upgrade to virtually all schemas.
As mentioned on the issue, [Azure Cloud Storage](https://learn.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata) does NOT support trailing dot (.) and hence similar shortcoming are imposed on Storage API.
There are few solutions that can be implemented to address shortcoming in Azure Cloud Storage, but none of those solutions are trivial and require lot of refactoring of common code or low level cloud storage classes on Storage service.
The specific issue only happens when record with and without trailing dot (.) are used on same PUT API call. If they are split in different requests then we can process such record-ids. Moreover, usage of such record-ids are only observed in reference-schema's OSDU bootstrapped data and it's not widely used.
This changeset loosen up previous [validation ](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/825) that outright reject all requests with trailing dot (.). It will still be a breaking change if users have automated workflow to create records that includes record-ids with and without trailing dot (.) on same request, but it will not completely drop the support the record-id with trailing dot (.).M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/825Reject record ids with unsupported characters at the end2024-01-31T23:00:32ZNeha KhandelwalReject record ids with unsupported characters at the endRelated issue #213.
For Storage create/update record API, if a record ID ends with a dot (.), a forward slash (/), or a backslash (\\) the data block for the record is not properly uploaded to the Microsoft storage account. Create/updat...Related issue #213.
For Storage create/update record API, if a record ID ends with a dot (.), a forward slash (/), or a backslash (\\) the data block for the record is not properly uploaded to the Microsoft storage account. Create/update record API should reject record IDs that end in these unsupported characters (i.e. return 400 bad request when such record IDs are used).M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/796Update persistablereference by UnitOfMeasureID2024-01-29T16:35:08ZKonrad KrasnodebskiUpdate persistablereference by UnitOfMeasureIDChanges related to [Issue](https://community.opengroup.org/osdu/platform/system/storage/-/issues/188 "Normalizer: meta[].unitOfMeasureID shouldbe preferred unit declaration")
Example record before persistableRereference update:
```plai...Changes related to [Issue](https://community.opengroup.org/osdu/platform/system/storage/-/issues/188 "Normalizer: meta[].unitOfMeasureID shouldbe preferred unit declaration")
Example record before persistableRereference update:
```plaintext
{
"records": [
{
"data": {
...
},
"meta": [
{
"kind": "osdu",
"name": "m",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":1.0,\"c\":1.0,\"d\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"L\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"unitOfMeasureID": "osdu:reference-data--UnitOfMeasure:ft:",
"propertyNames": ["VerticalMeasurements[].VerticalMeasurement"]
}
],
...
}
}
```
Example record after persistableRereference update and before unit conversion:
```plaintext
[
{
"recordJsonObject": {
"data": {
...
},
"meta": [
{
"kind": "osdu",
"name": "m",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.3048,\"c\":1.0,\"d\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"L\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"unitOfMeasureID": "osdu:reference-data--UnitOfMeasure:ft:",
"propertyNames": ["VerticalMeasurements[].VerticalMeasurement"]
}
],
...
}
]
```
WIP: integration tests needs to be added/updatedM22 - Release 0.25https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/751Adding the correlationId appended by a random uuid as the messageId in Record...2024-01-29T13:30:53ZSabarish K R EAdding the correlationId appended by a random uuid as the messageId in RecordChangedMessages, to enable traceabilityAdding the correlationId appended by a random uuid as the messageId in RecordChangedMessages, to enable traceabilityAdding the correlationId appended by a random uuid as the messageId in RecordChangedMessages, to enable traceabilityhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/834[MSCOSDU-2064] Added gcs version paths validation for update record flow2024-01-29T04:40:27ZDeepa Kumari[MSCOSDU-2064] Added gcs version paths validation for update record flowAs per #178, the GCS version array size grows huge causing issues. It was required to not allow update of a record where gcs array size is greater than a configurable specific soft limit. This change is sitting behind a feature flag.
Th...As per #178, the GCS version array size grows huge causing issues. It was required to not allow update of a record where gcs array size is greater than a configurable specific soft limit. This change is sitting behind a feature flag.
The implementation will ignore the records with greater versions than the soft limit and a warning will be logged in application logs.M23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/252FoR(Frame-Of-Reference) fixes - ibm2024-01-27T06:33:11ZRitika KaushalFoR(Frame-Of-Reference) fixes - ibmissue #55
This fix is for FoR (Frame-of-Reference) ,basically a header value needs to be passed with storage /query/records:batch endpoint to enable conversion of data. The mandatory values in the system to initiate this are :
1. Head...issue #55
This fix is for FoR (Frame-of-Reference) ,basically a header value needs to be passed with storage /query/records:batch endpoint to enable conversion of data. The mandatory values in the system to initiate this are :
1. Header in request : frame-of-reference=units=SI;crs=wgs84;elevation=msl;azimuth=true north;dates=utc;
1. CRS_API value in storage service at environment level which should point to CRS-CONVESRION-SERVICE Example: `CRS_API=https://<<csp-api-gateway>>/api/crs/converter/v2`
1. A variable `createCrsJWTToken=false` ha been added in CrsConversionService.java in storage core to stop regeneration of JWTToken and to pass the original header as is. This is kept for making it non breaking for CSP. In case CSP wants to pass same authorization header, the value can be false else default value of true results in new token generated as per CSP specific implementation of generating id token.
1. All test cases which were ignored earlier in PostFetchRecordsIntegrationTests.java have been enabled. Configuration discussed in 2 and 3 need to considered for passing the test cases.M8 - Release 0.11Ritika KaushalRitika Kaushalhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/835Resolve AWS internal scanned CVE2024-01-26T14:11:45ZBruce JinResolve AWS internal scanned CVEResolve AWS internal trivy scan CVE, will need to skip `logback` related CVEs since these need to be solved after Spring 6.0 upgradeResolve AWS internal trivy scan CVE, will need to skip `logback` related CVEs since these need to be solved after Spring 6.0 upgradeM23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/740Migration to JDK 17 (GONRG-7527)2024-01-25T13:47:33ZRiabokon Stanislav(EPAM)[GCP]Migration to JDK 17 (GONRG-7527)## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] I...## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Does this introduce a breaking change?
- [YES]
* Migration to Java 17.
* Unit test fixes with build run args and dependency upgrades.
* Refactored with Apache's `CloseableHttpClient` supporting all HTTP methods (including PATCH).M20 - Release 0.23Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/829Fix vulnerbilities2024-01-23T14:38:11ZBruce JinFix vulnerbilitiesFix existing critical CVEs and other Vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/issues/196
https://community.opengroup.org/osdu/platform/system/storage/-/issues/194
https://community.opengroup.org/osd...Fix existing critical CVEs and other Vulnerabilities:
https://community.opengroup.org/osdu/platform/system/storage/-/issues/196
https://community.opengroup.org/osdu/platform/system/storage/-/issues/194
https://community.opengroup.org/osdu/platform/system/storage/-/issues/197
https://community.opengroup.org/osdu/platform/system/storage/-/issues/198
https://community.opengroup.org/osdu/platform/system/storage/-/issues/199
https://community.opengroup.org/osdu/platform/system/storage/-/issues/200
https://community.opengroup.org/osdu/platform/system/storage/-/issues/201
https://community.opengroup.org/osdu/platform/system/storage/-/issues/202
https://community.opengroup.org/osdu/platform/system/storage/-/issues/203M23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/824AWS collaboration header support2024-01-23T14:35:50ZMark ChanceAWS collaboration header support# Merge request template# Merge request templateM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/827remove aws integration test override2024-01-22T17:20:54ZYunhua Koglinremove aws integration test override# Merge request template# Merge request templateM23 - Release 0.26Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/694pass data authz if user belongs to users.data.root group2024-01-22T13:54:52ZMingyang Zhupass data authz if user belongs to users.data.root group# Merge request template
https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/123# Merge request template
https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/123M19 - Release 0.22Mingyang ZhuMingyang Zhuhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/819Add support more than 10 patch operations in a single request2024-01-19T16:12:19ZYurii KondakovAdd support more than 10 patch operations in a single request## Type of change
- [x] Bug Fix
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
Currently,...## Type of change
- [x] Bug Fix
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
Currently, we have a limit of 10 operations on the maximum number of patch operations due to limitations on the part of Azure Cosmos DB.
[Partial document update in Azure Cosmos DB](https://learn.microsoft.com/en-us/azure/cosmos-db/partial-document-update)
For the API consumers, this number is less by 2, i.e. 8, as we add two more patch operations internally (at storage service).
The essence of the changes is that if the number of operations is more than 10,then we divide them into portions of a maximum of 10 operations.
This merge request is a second part of Bug fix. The first part is changes in the core-lib-azure library
https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/321M23 - Release 0.26Yurii KondakovYurii Kondakovhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/828[MSCOSDU-1979] fix code coverage2024-01-18T09:10:28ZDeepa Kumari[MSCOSDU-1979] fix code coverageIncrease the code coverage in storage service from 67% to 91%Increase the code coverage in storage service from 67% to 91%M23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/818Gonrg 9065 storage bootstrap disable2024-01-17T09:44:08ZAliaksandr Ramanovich (EPAM)Gonrg 9065 storage bootstrap disable
GC changes only
Disable storage bootstrap due to duplicating this process in Data bootstrap repo
GC changes only
Disable storage bootstrap due to duplicating this process in Data bootstrap repoM23 - Release 0.26Aliaksandr Ramanovich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/813[MSCOSDU-1894] upgrade reactor-netty, log4j2, okhttp and okio2024-01-12T00:21:10ZVidyaDharani Lokam[MSCOSDU-1894] upgrade reactor-netty, log4j2, okhttp and okio# Change details
### Common code
* upgraded `log4j2` version to `2.22.0`
### Azure
* upgraded `reactor-netty` related dependencies to version `1.1.14`
* upgraded `nimbus-jose-jwt-azure` to `9.30.2`.
* remediate `container_scanning` f...# Change details
### Common code
* upgraded `log4j2` version to `2.22.0`
### Azure
* upgraded `reactor-netty` related dependencies to version `1.1.14`
* upgraded `nimbus-jose-jwt-azure` to `9.30.2`.
* remediate `container_scanning` flagged vulnerabilities `log4j2,okio,okhttp`
# Changes in:
* [x] GCP
* [x] Azure
* [x] AWS
* [x] IBMM23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/793Added /liveness_check (GONRG-9039)2024-01-04T14:52:35ZRiabokon Stanislav(EPAM)[GCP]Added /liveness_check (GONRG-9039)## Type of change
- [ ] Bug Fix
- [X] Feature
https://community.opengroup.org/osdu/platform/system/storage/-/issues/191
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider ...## Type of change
- [ ] Bug Fix
- [X] Feature
https://community.opengroup.org/osdu/platform/system/storage/-/issues/191
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GC
- [x] IBM
## Does this introduce a breaking change?
- [NO]
## What is the new/expected behavior?
Added /liveness_check
## Have you added/updated Unit Tests and Integration Tests?
YesM23 - Release 0.26Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/814add liveness_check endpoint for azure2024-01-04T07:34:36Zsaketh somarajuadd liveness_check endpoint for azure- MR Reference: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/793
- Added `liveness_check` endpoint in collaboration filter for azure.- MR Reference: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/793
- Added `liveness_check` endpoint in collaboration filter for azure.M23 - Release 0.26VidyaDharani LokamVidyaDharani Lokam