Storage issueshttps://community.opengroup.org/osdu/platform/system/storage/-/issues2022-08-15T16:12:48Zhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/135Storage release/0.15 build Failure2022-08-15T16:12:48ZShrikant GargStorage release/0.15 build FailureStorage release/0.15 build Failure start failing as it is referring to core-common 0.15.0-SNAPSHOT which is cleaned up. Ideally SNAPShot versions should be be referenced.
So upgrading it to latest version is recomendedStorage release/0.15 build Failure start failing as it is referring to core-common 0.15.0-SNAPSHOT which is cleaned up. Ideally SNAPShot versions should be be referenced.
So upgrading it to latest version is recomendedM12 - Release 0.15Shrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/system/storage/-/issues/134Storage and PUT - Any way to work around the limit of 500 records?2022-12-09T13:35:40ZDebasis ChatterjeeStorage and PUT - Any way to work around the limit of 500 records?I was trying to persist standard reference values for entity such as UnitOfMeasure and hit this limit.
```
{
"code": 400,
"reason": "Validation error.",
"message": "createOrUpdateRecords.records: Up to 500 records can be ing...I was trying to persist standard reference values for entity such as UnitOfMeasure and hit this limit.
```
{
"code": 400,
"reason": "Validation error.",
"message": "createOrUpdateRecords.records: Up to 500 records can be ingested at a time"
}
```
Is there something we can do for work around (apart from having to split the original JSON load manifest into smaller chunks)?
cc - @krveduru for informationhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/133[BUG] Create error messages based on response from OPA2022-09-07T21:28:20ZRostislav Vatolinvatolinrp@gmail.com[BUG] Create error messages based on response from OPAStorage has failing tests due to incorrect logic related to integration with OPA. Storage should not be responsible for creating a custom error message in case OPA returns error. Please use error message returned from OPA. Please make su...Storage has failing tests due to incorrect logic related to integration with OPA. Storage should not be responsible for creating a custom error message in case OPA returns error. Please use error message returned from OPA. Please make sure all integration tests are passing when opa is turned on.
Related MR: https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/merge_requests/122Rostislav Vatolinvatolinrp@gmail.comRostislav Vatolinvatolinrp@gmail.comhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/132CORS blocking query/records:bacth endpoint2022-11-11T13:38:55ZYifan YeCORS blocking query/records:bacth endpointCORS does not allow frame-of-reference header and query/records:bacth endpoint is blocked by CORS.CORS does not allow frame-of-reference header and query/records:bacth endpoint is blocked by CORS.M14 - Release 0.17Yifan YeYifan Yehttps://community.opengroup.org/osdu/platform/system/storage/-/issues/131No update notification sent2022-08-23T21:00:11ZQiang FuNo update notification sentStep to reproduce:
1) setup notification endpoint
2) subscribe to "recordchange" topic
3) create a record by using api/storage/v2/records endpoint. Verify a "create" notification received.
4) modify the payload used in step 3 and run ap...Step to reproduce:
1) setup notification endpoint
2) subscribe to "recordchange" topic
3) create a record by using api/storage/v2/records endpoint. Verify a "create" notification received.
4) modify the payload used in step 3 and run api/storage/v2/records again, Another "create" notification received.
5) modify the payload used in step 4 and run api/storage/v2/records/?skipdupes=true, "create" notification received.
In step 4 and 5 we are expecting a "update" notificationhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/130Storage PUT: setting a non-number value to a number attribute results in an e...2022-08-23T15:45:28ZAn NgoStorage PUT: setting a non-number value to a number attribute results in an empty 400 response (no error message)For example, given this payload. This was provided:
` "value": Infinity`
```
curl --location --request PUT 'https://domain.com/api/storage/v2/records' \
--header 'accept: application/json' \
--header 'data-partition-id: osdu' \
--hea...For example, given this payload. This was provided:
` "value": Infinity`
```
curl --location --request PUT 'https://domain.com/api/storage/v2/records' \
--header 'accept: application/json' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--header 'Authorization: <token>' \
--data-raw '[
{
"acl": {
"owners": [
"data.default.owners@domain.com"
],
"viewers": [
"data.default.viewers@domain.com"
]
},
"data": {
"ExtensionProperties": {
"osdu": {
"curvesProperties": [
{
"curveID": "CTEM_GPITF",
"properties": [
{
"name": "MEASURE-POINT-OFFSET",
"value": Infinity
}
]
}
]
}
}
},
"kind": "osdu:wks:work-product-component--WellLog:1.1.0",
"legal": {
"legaltags": [
"osdu-default-legal"
],
"otherRelevantDataCountries": [
"US"
]
}
}
]'
```
Reponse:
Empty 400
![image](/uploads/18749c6ea879c9c888a3c5c173288b23/image.png)https://community.opengroup.org/osdu/platform/system/storage/-/issues/129Storage returns inconsistent and wrong responses if nested attribute filters ...2023-07-07T12:15:36ZAn NgoStorage returns inconsistent and wrong responses if nested attribute filters which do not exist are specifiedUsing /api/storage/v2/records/{id}
optional attribute filter:
![image](/uploads/05e101feb2102c16adba4575f31a4aa7/image.png)
Example, given this data:
```
"data": {
"relationships": {
"well": {
"id": "slb-osdu-tryme:...Using /api/storage/v2/records/{id}
optional attribute filter:
![image](/uploads/05e101feb2102c16adba4575f31a4aa7/image.png)
Example, given this data:
```
"data": {
"relationships": {
"well": {
"id": "slb-osdu-tryme:well",
"name": "Card Creek 2"
},
"relatedItems": {
"ids": [
"Log1",
"Marker1"
],
"names": [
"Log Name1",
"Marker Name1"
]
}
}
```
A few observances when filtering with string:
data.something returns 200
data.relationships.something returns 200
data.relationships.well.something returns 200
data.relationships.well.number returns 500 at first. Then after a few tries, it returns 200.
data.relationships.something returns 200
data.relationships.relatedItem.something returns 500 (no s on relatedItems)
data.relationships.relatedItems.something returns 500 at first, then 200 after that.
Expected return code should be 400.M19 - Release 0.22https://community.opengroup.org/osdu/platform/system/storage/-/issues/127Soft-deleted record was skipped when re-ingested with same data2022-08-23T15:54:49ZAn NgoSoft-deleted record was skipped when re-ingested with same data**Steps to reproduce the current behavior:**
1. Ingest a record
2. Soft-delete the record
3. Fetch the record to confirm it is now "inactive", "not found"
**Case 1:** Works as expected
4. Ingest the same record using the same id and D...**Steps to reproduce the current behavior:**
1. Ingest a record
2. Soft-delete the record
3. Fetch the record to confirm it is now "inactive", "not found"
**Case 1:** Works as expected
4. Ingest the same record using the same id and DIFFERENT data, skipdupes=true
> Record was NOT skipped. Deleted record became active again. A new version of the record is created.
> Example response:
```
{
"recordCount": 1,
"recordIds": [
"osdu:document:ee7e8869217541a8b31f4e2ea18f7e3a"
],
"skippedRecordIds": [],
"recordIdVersions": [
"osdu:document:ee7e8869217541a8b31f4e2ea18f7e3a:1654731042152281"
]
}
```
5. Soft-delete the record
6. Fetch the record to confirm it is now "inactive", "not found"
**Case 2:** Skips the record even though it was already deleted
7. Ingest the same record using the same id, SAME data, skipdupes=true
> Record was skipped. So the record remains "inactive", "not found". The PUT call did nothing to the record.
>Example response:
```
{
"recordCount": 1,
"skippedRecordIds": [
"slb-osdu-dev-sis-internal-hq:document:ee7e8869217541a8b31f4e2ea18f7e3a"
]
}
```
**Expected behavior:**
If skipdupes is true
- if the record doesn't exist at all, then create a new record.
- **if the record was soft-deleted, then make the record active again if the data is the same (last deleted version becomes the latest version), or create a new version if data is different.**
- if the record exists,
- if the data is the same, then skip it.
- if data is different, then create a new version
If skipdupes is false:
- if the record doesn't exist at all, then create a new record.
- **if the record was soft-deleted, then create a new version of the record**
- if the record exists, then a new version of the record will be created, regardless whether the data is the same or different.https://community.opengroup.org/osdu/platform/system/storage/-/issues/126All versions of a record have the same modifyUser and modifyTime2023-05-30T10:26:54ZAn NgoAll versions of a record have the same modifyUser and modifyTimeThe concept is that one record should have 1 version of metadata.
However, in regard to modifyUser and modifyTime attributes, they should be different for each version.
Currently, the behaviors are as implemented, but the behavior by th...The concept is that one record should have 1 version of metadata.
However, in regard to modifyUser and modifyTime attributes, they should be different for each version.
Currently, the behaviors are as implemented, but the behavior by the above concept is wrong.
So with the current behavior, for multiple versions of the same record modifyTime and modifyUser value are same and they are overwritten to all versions during every modification made to the record.
Which means for records having only 1 version, it is like below.
|version1|
|:-------|
|createUser|
|createTime|
But when the record is modified and multiple versions are created, the metadata of the record for latest version is applied to all versions including the first version as well, and all versions have value for modifyUser and modifyTime attributes.
|version1|version2 |version3|
|:-------|:--------|:--------|
|createUser| createUser| createUser|
|createTime| createTime| createTime|
|modifyUser2 |modifyUser2|modifyUser2|
|modifyTime2 |modifyTime2|modifyTime2|
**Expected:**
Version 1 should only have createUser and createTime. modifyUser and modifyTime should not exist in the first version.
Version 2+ should have different modifyUser and modifyTime for each version
|version1|version2 |version3|
|:-------|:--------|:--------|
|createUser| createUser| createUser|
|createTime| createTime| createTime|
| |modifyUser1|modifyUser2|
| |modifyTime1|modifyTime2|https://community.opengroup.org/osdu/platform/system/storage/-/issues/125Very high number of 429s on CosmosDb when there is a usage spike in Storage `...2023-07-19T19:35:51ZAlok JoshiVery high number of 429s on CosmosDb when there is a usage spike in Storage `query/records:batch api`In one of our client environments, we are consistently seeing very high number of 429 errors from CosmosDb. This is causing latency spikes for Storage apis.
From our investigation, this seems to be related to the query/records:batch api...In one of our client environments, we are consistently seeing very high number of 429 errors from CosmosDb. This is causing latency spikes for Storage apis.
From our investigation, this seems to be related to the query/records:batch api performance/optimization issue. We see a direct correlation between `query/records:batch api` spike and CosmosDb 429 error spike within multiple time windows. Please see attached images for reference.
In the first image, we can see a time window when CosmosDb threw a lot of 429 errors. In the second image, we can see Storage api usage pattern. Most of the api calls are made to the `query/records:batch api` which also affects latency numbers. The patterns on both images are very similar
![ComosDb_usage](/uploads/898f423082ae4193bb7636b058905555/ComosDb_usage.PNG)![api_usage](/uploads/ad233ce903b1596a3dc1f6048548088f/api_usage.PNG)
We've tried increasing the RUs on cosmosDb on multiple incidents but that doesn't help.
Further load tests showed that query/records:batch can be a root cause of the 429 errors.
Into the scope of the issue fixing it would be reasonable to implement some features from the topic https://docs.microsoft.com/en-us/azure/cosmos-db/sql/performance-tips-query-sdk?tabs=v3&pivots=programming-language-javahttps://community.opengroup.org/osdu/platform/system/storage/-/issues/124ADR: Supporting data block modification through Storage PATCH API call2023-07-05T09:41:17ZMandar KulkarniADR: Supporting data block modification through Storage PATCH API callSupporting data block modification Storage PATCH API call
## Status
- [X] Proposed
- [X] Trialing
- [X] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
Only record tags, legal tags and ACLs can up updated through PATCH AP...Supporting data block modification Storage PATCH API call
## Status
- [X] Proposed
- [X] Trialing
- [X] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
Only record tags, legal tags and ACLs can up updated through PATCH API in storage service.
The PATCH API cannot be used to update the data block in the record/s. For updating data block, PUT API needs to be used.
## Tradeoff Analysis
Updating an individual attribute inside data block needs two calls currently, one to GET the record and then PUT call to update the record content with new attribute value.
Providing PATCH API to update attributes in data block will support doing this operation in just one call to OSDU storage.
## Decision
We can update PATCH API to support modifications in data blocks. The API will continue to follow the [rfc6902 standard](https://www.rfc-editor.org/rfc/rfc6902.html).
Currently the PATCH API supports modifications in record tags, legal tags and ACLs only. It supports 3 operations namely add, replace and remove.
The same operations would be supported for data block.
- In "add" operation, specified property from the request would be appended with values provided in "value" field.
- In "replace" operation, specified property from the request would be fully replaced by values provided in "value" field.
- In "remove" operation, values provided in "value" field would be removed for specified property from the request.
Users specify the complete path to the property they want to update in "path" field, i.e. "/acl/viewers" indicates the values for metadata acl viewers would be updated.
Similarly "/data/TechnicalAssuranceID" would indicate that TechnicalAssuranceID attribute from the data block would be updated.
"/data/CurrentOperatorID" would indicate that CurrentOperatorID attribute from the data block would be updated.
"/data/EXtensionProperties/Attribute1" would indicate that Attribute1 from the ExtensionProperties inside data block would be updated.
"/data/SpatialLocation/SpatialGeometryTypeID" would indicate that SpatialGeometryTypeID from the SpatialLocation inside data block would be updated.
Version of the record would be incremented in case of data block update through PATCH API to maintain consistent behavior with PUT API.
## Consequences
- PATCH API behavior will be updated.
- Storage service documentation needs to be updated.M17 - Release 0.20https://community.opengroup.org/osdu/platform/system/storage/-/issues/123Storage GET record returns 404 for records with optional version (Record ID e...2023-06-06T20:04:08ZAn NgoStorage GET record returns 404 for records with optional version (Record ID ending with colon)Storage GET /api/storage/v2/records/{id} returns 404 error for records whose ID ends with a colon (version is empty).
For example, "osdu:master-data--Wellbore:nz-100000391126:"
This is the case where the version component is empty (this...Storage GET /api/storage/v2/records/{id} returns 404 error for records whose ID ends with a colon (version is empty).
For example, "osdu:master-data--Wellbore:nz-100000391126:"
This is the case where the version component is empty (this is allowed as part of [this change](https://community.opengroup.org/osdu/platform/system/storage/-/issues/26#summary-january-26-2021) in record ID validation).
Expected behavior should be returning the latest version of the record.https://community.opengroup.org/osdu/platform/system/storage/-/issues/122Storage Service Records Fetch Error2022-08-24T10:52:45ZSamiullah GhousudeenStorage Service Records Fetch Error**Not able to retrieve records from Storage Service **
If record id contains html encoded characters(%2F), then Storage service doesn't returns expected result but Search query returns as expected.
This issue same across all CSP's - A...**Not able to retrieve records from Storage Service **
If record id contains html encoded characters(%2F), then Storage service doesn't returns expected result but Search query returns as expected.
This issue same across all CSP's - AZURE, GCP, IBM & AWS.
For example , below query returns expected result through Search Service -
{
"kind": "*:wks:reference-data--UnitOfMeasure:1.0.0",
"limit": 10,
"aggregateBy":"kind",
"query":"id:\"osdu:reference-data--UnitOfMeasure:V%2FB\" OR id:\"opendes:reference-data--UnitOfMeasure:v%2Fv\" OR id:\"odesprod:reference-data--UnitOfMeasure:H%2Fm\" OR id:\"opendes:reference-data--UnitOfMeasure:US%2FF\""
}
However, in Storage service getting response - HTTP Status 400 – Bad Request
<!doctype html>
<html lang="en">
<head>
<title>HTTP Status 400 – Bad Request</title>
<style type="text/css">
body {
font-family: Tahoma, Arial, sans-serif;
}
h1,
h2,
h3,
b {
color: white;
background-color: #525D76;
}
h1 {
font-size: 22px;
}
h2 {
font-size: 16px;
}
h3 {
font-size: 14px;
}
p {
font-size: 12px;
}
a {
color: black;
}
.line {
height: 1px;
background-color: #525D76;
border: none;
}
</style>
</head>
<body>
<h1>HTTP Status 400 – Bad Request</h1>
</body>
</html>Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/system/storage/-/issues/121Storage Schema endpoints should be obsoleted2022-08-24T10:54:38ZGary MurphyStorage Schema endpoints should be obsoletedRemove code and config related to the storage schemas APIs from OSDU as they are EOL.
The following APIs are to be removed
- GET /Schema
- DELETE /Schema
- POST /schemaRemove code and config related to the storage schemas APIs from OSDU as they are EOL.
The following APIs are to be removed
- GET /Schema
- DELETE /Schema
- POST /schemahttps://community.opengroup.org/osdu/platform/system/storage/-/issues/118[Azure] Deletion records with invalid legaltag without any logging2022-04-12T14:49:06ZYauheni Lesnikau[Azure] Deletion records with invalid legaltag without any loggingThere is an message handler which process the `LegalTagChanged ` events, and if the legal tag became invalid, appropriate records should be mark as `Inactive` (soft deleted). The issue is that there is no any logging which explicitly say...There is an message handler which process the `LegalTagChanged ` events, and if the legal tag became invalid, appropriate records should be mark as `Inactive` (soft deleted). The issue is that there is no any logging which explicitly says that deletion performs there.Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/117Storage fails to delete large number of records upon legal tag expiration2024-03-21T15:19:58ZAn NgoStorage fails to delete large number of records upon legal tag expirationIf there are large number of records associated with a legalTag that expires after running the cron job, we are seeing availability issues and inconsistent result in terms of record searchability.
**Observations:**
**LegalTag cron job...If there are large number of records associated with a legalTag that expires after running the cron job, we are seeing availability issues and inconsistent result in terms of record searchability.
**Observations:**
**LegalTag cron job update issue:**
**Scenario**: I have a large number of records (in the 6 digits) that are associated with a legalTag (i.e. the record metadata has a particular legalTag (let's call it lt1) in the legal.legaltags section). The legalTag lt1 is set to expire soon
**Event**: lt1 expires
**Action 1** : Cron job `updateLegalTagStatus` is triggered on a periodic basis, which grabs the legalTags that have changed their state (valid to invalid and invalid to valid) and publishes this information onto SB topic 'legaltags' and EG topic 'legaltagschangedtopic'. The legalTag also changes its state in the CosmosDb
'legaltagschangedtopic' has an event subscription to SB topic 'legaltagschangedtopiceg', which has a subscription 'eg_sb_legaltagssubscription'
**Action 2 **: Storage service pulls messages from 'eg_sb_legaltagssubscription' for LegalTag update events and updates records associated with lt1. Storage updates the recordMetadata with active/inactive record status and publishes the change onto SB and EG for indexer-queue to consume.
**Expected outcome:** All records associated with lt1 are now inactive. They are unsearchable from Storage and Search APIs.
**Actual outcome:** Some records associated with lt1 are now inactive. They are unsearchable from Storage and Search APIs. I can still search other records.
**Issue**: Not all records are getting pulled from Storage service at **Action 2** to be processed. Thus, many records simply don't change their state, although the legalTag is invalid now.
**Observed behavior/possible improvements:**
1. The context of legalTag change (active to inactive or inactive to active) is not considered by Storage when fetching records to update. Storage tries to fetch ALL records for that legalTag with the query
SELECT * FROM c WHERE ARRAY_CONTAINS(c.metadata.legal.legaltags, lt1). In case of large number of records, this is a longer operation. We observed throttling on the cosmos-db during this process
2. No way to retry. Because Legal service updates the letalTag status in cosmosDb, running the `updateLegalTagStatus` job again will not pick up this legal tag. To do this, we are required to manually change the status of the legalTag and run the cron job again. Upon manual retries, we face the issue above where Storage is trying to process ALL records again.
3. What happens when Storage job is interrupted, possibly due to pod restart (high cpu utilization) or network error or cosmosDb error? Retrying the whole job doesn't help muchChad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/storage/-/issues/116In Azure environment the end point to query the data with limit is not working2022-08-26T11:59:00ZKamlesh TodaiIn Azure environment the end point to query the data with limit is not workingIn the Azure environment, the end point to query the data with a limit is not working.
e.g. https://osdu-ship.msft-osdu-test.org/api/storage/v2/query/kinds?limit=10
Response: 400 Bad Request
{
"code": 400,
"reason": "Limit not suppo...In the Azure environment, the end point to query the data with a limit is not working.
e.g. https://osdu-ship.msft-osdu-test.org/api/storage/v2/query/kinds?limit=10
Response: 400 Bad Request
{
"code": 400,
"reason": "Limit not supported",
"message": "The limit is invalid"
}
@debasisc @sehuboy @kumar_vaibav @ChrisZhangM10 Patch - Release 0.13 patchKrishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/system/storage/-/issues/114No audit log for succeed PATCH updates2022-03-24T14:46:56ZYauheni LesnikauNo audit log for succeed PATCH updatesWhen the PATCH update performs there is audit logging only for failed record ids. We need to add similar logging for the succeed ones as well.When the PATCH update performs there is audit logging only for failed record ids. We need to add similar logging for the succeed ones as well.M11 - Release 0.14Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/113Storage /records endpoint without having Content-type in the header throws 41...2023-03-01T04:46:31ZAn NgoStorage /records endpoint without having Content-type in the header throws 415 errorStorage /records endpoint without having Content-type in the header throws 415 error codeStorage /records endpoint without having Content-type in the header throws 415 error codehttps://community.opengroup.org/osdu/platform/system/storage/-/issues/112Potential defect related to Delete methods due to no response body - Storage ...2022-11-21T10:12:01ZJevon WilliamsPotential defect related to Delete methods due to no response body - Storage Delete RecordDELETE API Methods that has no response body. This call will provide both a 204 success code and a 404 failure code but it does not provide any details and/or explanations:
Example 204 code is returned but it does not say item has s...DELETE API Methods that has no response body. This call will provide both a 204 success code and a 404 failure code but it does not provide any details and/or explanations:
Example 204 code is returned but it does not say item has successfully deleted.
Example 404 code is returned but it does not say why the error code was returned
URL Endpoint - base_url/api/storage/v2/records/{{recordIds}}![deleteAPI_Storage_Record_screenshot](/uploads/396a120b7291f242bf4a0fac9b1b12ed/deleteAPI_Storage_Record_screenshot.PNG)