Storage issueshttps://community.opengroup.org/osdu/platform/system/storage/-/issues2024-03-07T13:08:37Zhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/179Storage batch API returns 404 for unauthorized records2024-03-07T13:08:37ZAn NgoStorage batch API returns 404 for unauthorized records**Use-case:** Reindex Kind API is called.
Noted in the logs there were 404s returned.
Record Fetch on some of the impacted records, 403s were returned.
Investigation shows Batch Record fetch returned 404s instead.
Issue identified f...**Use-case:** Reindex Kind API is called.
Noted in the logs there were 404s returned.
Record Fetch on some of the impacted records, 403s were returned.
Investigation shows Batch Record fetch returned 404s instead.
Issue identified from this workflow:
- Storage batch API responds unauthorized records (403) as not found (404)
### ADR: Storage batch API responds unauthorized records (403) as not found (404)
#### Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
#### Context & Scope
The current behavior of Storage batch API: if a record is not authorized, it is put in the _notFound_ field of the response body along with other not found records. The response body in this case looks like this:
```
{
"records": [],
"notFound": [
"opendes:facet:unauthorizedrecord1",
"opendes:facet:unauthorizedrecord2",
//other not found records...
],
"conversionStatuses": []
}
```
#### Solution
To fix this behavior of the Storage batch API we can introduce a new field to the response body. The proposed solution is to add a new field (_unauthorized_) to the response body, so we can distinguish between unauthorized and actual not found records. Sample response body:
```
{
"records": [],
"notFound": [
//not found records...
],
"unauthorized": [
"opendes:facet:unauthorizedrecord1",
"opendes:facet:unauthorizedrecord2"
],
"conversionStatuses": []
}
```
#### Сonsequence
This solution is a breaking change as it implies changing API contract. It will include a change in the core library, a change in Storage, and then a change in the Indexer service to handle batch API response.Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/storage/-/issues/178ADR: CosmosDb saturation/throttling when records reach too many versions2024-03-25T06:43:30ZAlok JoshiADR: CosmosDb saturation/throttling when records reach too many versions## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
***ISSUE***: Storage service stability issues due to too many versions of records.
***User behavior that causes this issue***: ...## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
***ISSUE***: Storage service stability issues due to too many versions of records.
***User behavior that causes this issue***: Creating a lot of versions for the same record ID. When multiple applications/teams do this long enough, we have too many versions for many records. There are no checks in place to prevent this scenario. We eventually hit infrastructure limits (i.e. CosmosDb document max size 2MB) but observe service instability much before.
***Why is this a problem***: Record versions are stored as part of record metadata. This is part of the `gcsVersionPaths` array. Each version is a string that represents the full path to the version's blob location. Record metadata is stored in CosmosDb. While CosmosDb has a hard size limit (2MB) for each document, this size is already too big when RU usage is considered. If we have hundreds or thousands of such records being updated, the total RU consumed is very high, incurring huge costs. This scenario poorly impacts service latency and availability. While not ideal, it is quite possible for applications to create versions of the same record for their workflows.
![image](/uploads/3f53fa471e7566a04d69ea539712db76/image.png)
For reference, here are some preliminary observations on the number of versions, size of the document and RU consumed to perform an UPSERT on a ***single*** document (note that the number of versions is not an ***absolute*** indicator to say how much RU will be consumed in performing an UPSERT, because its the size of the document that matters, and each version string can be of different length. One can fit a lot more versions if each version's length is small. However, as we stand today, it is the only metadata property that is causing documents to be big).
~1500 versions, ~300 RU consumed, ~243kb file size
~1500 versions, ~370 RU consumed, ~300kb file size
~3800 versions, ~1250 RU consumed, ~750kb file size
~5300 versions, ~1253 RU consumed, ~880kb file size
~9850 versions, ~2502 RU consumed, ~1.3mb file size
It is quite easy to have a few hundred or thousand records cripple the system once the records reach certain number of versions.
***CLARIFICATION***: The issue we observed is more specific to the Azure use case. Infrastructure limitations (i.e. cost to access a large document, hard limit on the size of the document) may vary per CSP (i.e. 2MB for CosmosDb, 1MB for GCP datastore). Other CSPs may see this issue once the number of versions reaches a certain number.
## Tradeoff Analysis
It is clear we want to limit the number of record versions. We see 2 ways to achieve this.
1. ***Set a hard limit*** on the number of versions on each record (say 1000) (preferred approach).
- Pros: Easy to implement, no behind-the-scenes magic.
- Cons: Breaking change for the existing workflows, when their records already have more than 1000 versions. Needs advance notice of breaking change and time for teams to update the workflows.
We can roll this out by first introducing a `deleteVersion` API in Storage that would give users time to delete older versions by themselves before breaking change is introduced so they don't break immediately.
2. ***Only keep 1000 recent versions***. For new records, this would mean actively start deleting the oldest version once we reach 1000 versions. For existing records with more than 1000 versions, this would mean cleaning up all older versions.
- Pros: Older versions are cleaned up for users automatically.
- Cons: Still a breaking change as older versions would get deleted automatically. Involves behind-the-scenes cleanup of older versions. For records that currently have more than 1000 records, this includes all remaining versions. There can be failure scenarios with cleanup and performance implications.
## Consequences
Storage will introduce a limit on the number of versions a record can have. Depending on the solution we choose, API will either fails after n versions (hard limit) OR older versions will get deleted automatically.M23 - Release 0.26Alok JoshiChad LeongThulasi Dass SubramanianOm Prakash GuptaAlok Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/issues/177Integration test coverage for users.data.root2023-07-20T11:05:00ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comIntegration test coverage for users.data.rootChanges to data authentication were recently introduced with the merge request: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/694. However, we currently lack integration test cases to cover these modificat...Changes to data authentication were recently introduced with the merge request: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/694. However, we currently lack integration test cases to cover these modifications.
It is essential to ensure that these changes won't disrupt the current flow and that `users.data.root` will consistently have access to ingested data.
To address this, we need to implement integration test cases to cover the new data authentication mechanisms.M20 - Release 0.23Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/176Storage x-collaboration header bug2023-09-26T14:21:44ZShane HutchinsStorage x-collaboration header bugFound this issue in /api/storage/v2/query/records, /api/storage/v2/query/records:batch
Received a response with 5xx status code: 500
Run this curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H ...Found this issue in /api/storage/v2/query/records, /api/storage/v2/query/records:batch
Received a response with 5xx status code: 500
Run this curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' -H 'x-collaboration: ^À' 'https://osdu.r3m18.preshiptesting.osdu.aws/api/storage/v2/query/records?kind='
curl -X POST -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' -H 'x-collaboration: ^À' -d '[]' https://osdu.r3m18.preshiptesting.osdu.aws/api/storage/v2/records/delete
PUT /api/storage/v2/records
curl -X PUT -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' -H 'x-collaboration: €' -d '[]' https://osdu.r3m18.preshiptesting.osdu.aws/api/storage/v2/records
Azure PUT /api/storage/v2/records:
curl -X PUT -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' -H 'x-collaboration: €' -d '[]' https://osdu-ship.msft-osdu-test.org/api/storage/v2/records
Confirmed this bug in AWS and Azure.https://community.opengroup.org/osdu/platform/system/storage/-/issues/173Does not detect mismatch of entity name between "kind" and "id"2023-06-06T00:42:59ZDebasis ChatterjeeDoes not detect mismatch of entity name between "kind" and "id"I made this test case to create record directly by using Storage service and then the same record by using Manifest-based Ingestion.
```
"kind": "osdu:wks:work-product-component--TubularComponent:1.0.0",
"id": "osdu:work...I made this test case to create record directly by using Storage service and then the same record by using Manifest-based Ingestion.
```
"kind": "osdu:wks:work-product-component--TubularComponent:1.0.0",
"id": "osdu:work-product-component--TubularAssembly:TUBULARDC31May",
```
As you can see "kind" speaks of **TubularComponent** whereas "id" speaks of **TubularAssembly**.
Storage service seems very forgiving. It creates the record and also Indexer replicates the record in Index store. So, we can also retrieve by using Search service.
Whereas Manifest-based Ingestion rejects this JSON payload wit suitable reason, as expected.
See this document.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M17/Test_Plan_Results_M17/Core_Services/M17-AWS-Storage-service-test-sanity.docxhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/172Metadata update API succeeds on remove operation on a `tag` if the tag doesn'...2023-05-25T10:36:21ZAlok JoshiMetadata update API succeeds on remove operation on a `tag` if the tag doesn't existSteps to reproduce:
- Create a record with some tags
- Try to update the record metadata via [metadata update API](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#metadata-updat...Steps to reproduce:
- Create a record with some tags
- Try to update the record metadata via [metadata update API](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#metadata-update-api) by removing a non-existing tag
```
curl --request PATCH \
--url '/api/storage/v2/records' \
--header 'accept: application/json' \
--header 'authorization: Bearer <JWT>' \
--header 'content-type: application/json'\
--header 'Data-Partition-Id: common'
--data-raw ‘{
"query": {
"ids": [
"tenant1:type:unique-identifier:version"
]
},
"ops": [
{
"op":"remove",
"path":"/tags",
"value":[
"tagthatdoesntexist"
]
}
]
}
```
This should return 4xx, but returns 2xxhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/171Metadata only updates (via PATCH api) creates a mismatch in modifyUser and mo...2023-07-05T09:50:37ZAlok JoshiMetadata only updates (via PATCH api) creates a mismatch in modifyUser and modifyTime fields between record metadata and record data[This ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/148) introduces separate modifyTime and modifyUser fields for every version of OSDU Storage record. This creates a mismatch between modifyTime and modifyUse...[This ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/148) introduces separate modifyTime and modifyUser fields for every version of OSDU Storage record. This creates a mismatch between modifyTime and modifyUser fields for metadata and data objects respectively.
Repro steps:
- Create a storage record
- Modify the metadata ACL with PATCH api
- Retrieve the record with Storage records:batch api or getRecord api
- modifyTime and modifyUser fields are not returned.
OR
- Create a storage record
- Update the same record with PUT api
- Modify the metadata ACL with PATCH api
- Retrieve the record
- modifyTime and modifyUser are returned but not correct
Expected: From a user's perspective, when they update a record (either metadata or data or both), they should get back modifyUser and modifyTime values appropriatelyhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/170Invalidate derived data when parent record is deleted2023-03-31T10:02:02ZAn NgoInvalidate derived data when parent record is deletedDerived data (records with ancestry/parent) inherit the legal tags from the parent record(s).
So when at least one of the parent records is deleted, then the children records are no longer valid. Without this step, there are records wit...Derived data (records with ancestry/parent) inherit the legal tags from the parent record(s).
So when at least one of the parent records is deleted, then the children records are no longer valid. Without this step, there are records with invalid legal tags (or no legal tag) still exists in the system.https://community.opengroup.org/osdu/platform/system/storage/-/issues/169ADR: API to purge a batch of storage records2023-05-02T12:16:58ZMandar KulkarniADR: API to purge a batch of storage recordsNew API in Storage service to purge a batch of records
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The OSDU Storage service provides 2 ways to delete a record. One way is ...New API in Storage service to purge a batch of records
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The OSDU Storage service provides 2 ways to delete a record. One way is to logically delete the record in which the record with same id can be revived later because its version history is maintained.The other way is to permanently delete the record (called as purging) in which case, the record's version history is deleted too. This operation cannot be undone meaning records purged cannot be revived.
In both types of deletions, the record content cannot be accessed using storage or search service.
The storage service provides separate APIs for logical deletion (`POST /records/{id}:delete`) and purging of records (`DELETE /records/{id}`).
The storage service provides API for logical deletion of batch of records (`POST /records/delete`), but such an API is not available for purging of records.
The proposal is to provide an API on storage service to support purging of batch of records, where the maximum batch of size 500 will be supported.
Only the record IDs passed in the request body will be deleted not including any linked records or files if they exist. Cleaning up of all the linked records, such as child records, records in relationship block, and actual data (files ingested via workflow service), would not be in the scope of this API. It would be the user's responsibility.
The new bulk API will work on active as well as non-active (soft-deleted) records, similar to the existing purge API.
Purging of records can be performed by the owner of the records and the owner should be part of users.datalake.admins group.
The API response would be similar to the response of the logical deletion API that is `POST /records/{id}:delete`
In case of partial success, the response code would be 207 and the not-deleted-record-IDs would be listed in the response.
## Tradeoff Analysis
In the absence of an API to purge a batch of records, users would have to call the DELETE API once for every record and it would increase the number of calls to the storage service.
## Decision
Provide an admin-only API to purge a batch of records, with maximum batch size of 500 records.
The Open API specs for storage service with new API is here:
[storage_openapi_batchpurge.yaml](/uploads/1da3f68253419edd693a87d706049565/storage_openapi_batchpurge.yaml)
## Consequences
- New API on Storage service would be available.
- Documentation of Storage service should be modified with details for the new API.https://community.opengroup.org/osdu/platform/system/storage/-/issues/168Storage should allow empty data block upon record creation/update2023-03-22T04:13:47ZAn NgoStorage should allow empty data block upon record creation/updateStorage PUT api should allow empty data block upon record creation/update if that is compliant with the schema being defined.
Currently, data block is required.
data: {}
This is a breaking change since it changes the behavior of the ...Storage PUT api should allow empty data block upon record creation/update if that is compliant with the schema being defined.
Currently, data block is required.
data: {}
This is a breaking change since it changes the behavior of the API.
Indexer service needs to be checked to ensure empty data block is being handled correctly.https://community.opengroup.org/osdu/platform/system/storage/-/issues/166Need example of how to use the POST /query/records:batch Fetch multiple rec...2023-04-20T03:00:55ZKamlesh TodaiNeed example of how to use the POST /query/records:batch Fetch multiple recordsThe Storage API documentation mention about
POST /query/records/batch Fetch multiple records. Would like to get the sample of how is this feature expected to be used.
Need clarification on
Account ID is the active OSDU account (OSDU ...The Storage API documentation mention about
POST /query/records/batch Fetch multiple records. Would like to get the sample of how is this feature expected to be used.
Need clarification on
Account ID is the active OSDU account (OSDU account or customer's account) which the users choose to use with the Search API.
frame-of-reference: This value indicates whether normalization applies, should be either 'none' or 'units=SI;crs=wgs84;elevation=msl;azimuth=true north;dates=utc;'
@chad @debasiscM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/storage/-/issues/163The request to get records of particular kind using the limit is not working.2023-06-20T05:07:07ZKamlesh TodaiThe request to get records of particular kind using the limit is not working.The Storage API CI/CD v1.11 (from Platform Validation project) was working on all the platforms and passing with 100% pass rate.
https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/12_CICD_Setup_Stor...The Storage API CI/CD v1.11 (from Platform Validation project) was working on all the platforms and passing with 100% pass rate.
https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/12_CICD_Setup_StorageAPI/Storage%20API%20CI-CD%20v1.11.postman_collection.json
At present, it is still passing with 100% pass rate in AWS R3 M16 Platform Validation (forum testing environment)
But it is not passing with 100% pass rate in all other Platform Validation CSPs environments as well as
it is not passing with 100% pass rate in all CSPs environments in pre-ship
In the referenced collection Request #8 is failing.
The following request for STORAGE API is in question 08 - Storage - Get all records for a kind with limit of 10 records
=====================================================================
e.g. of passing in Platform Validaition R3 M16 (forum testing)
curl --location 'https://r3m16.forumtesting.osdu.aws/api/storage/v2/query/records?limit=10&kind=osdu%3Awks%3AautoTest_955280%3A1.1.0' \
--header 'data-partition-id: osdu' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer eyJraWQiOi...4XnucQETfnB3biA' \
--header 'Cookie: session=eyJfZnJlc2giOmZhbHNlLCJfcGVybWFuZW50Ijp0cnVlfQ.Y_VNrw.SMJbZoZwlkMYCD7E9ge4ICPnqJY'
https://{{STORAGE_HOST}}/query/records?limit=10&kind={{authority}}:{{schemaSource}}:{{entityType}}:{{schemaVerMajor}}.{{schemaVerMinor}}.{{schemaVerPatch}}
The response code: 200 OK
{
"results": [
"osdu:999611481173:999301114394"
]
}
===================================================================
Example of when it is failing
curl --location 'https://r3m16-ue1.preshiptesting.osdu.aws/api/storage/v2/query/records?limit=10&kind=osdu%3Awks%3AautoTest_20923%3A1.1.0' \
--header 'data-partition-id: osdu' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer eyJraWQiOi...tW7kPscDabFJ3sEPeNA'
Response code: 415 Unsupported Media Type
Body of response is blank
It is same message for all the CSP where failure is happening
============================================================================
@chad @debasiscM16 - Release 0.19https://community.opengroup.org/osdu/platform/system/storage/-/issues/162Record ACL should be case insensitive2023-03-09T18:17:51ZAn NgoRecord ACL should be case insensitiveEntitlements group creation always lowercases the group name, regardless of the input.
Storage honors the ACL group name case sensitivity. This creates inconsistency for ACL validation.
**For example:**<br>
User creates a data group cal...Entitlements group creation always lowercases the group name, regardless of the input.
Storage honors the ACL group name case sensitivity. This creates inconsistency for ACL validation.
**For example:**<br>
User creates a data group called: data.SomeGroup.viewers<br>
Upon this request, Entitlements creates a group called: data.somegroup.viewers
Upon creating a record, the user enters data.SomeGroup.viewers as the ACL.<br>
If the user tries to fetch the record, a 403 is returned since Entitlements only sees group data.somegroup.viewers.
**Fix:**<br>
**For existing records (addressing the ghosted records):** Storage fetch record validation should lowercase the ACL group against the list of groups returned from Entitlements.<br>
**Long term solution:** The fix should be in the record creation. Storage PUT API should lowercase the ACL upon record creation. OR We could fail the PUT request if the ACL group has mixed case. Note that there is no ACL group existence validation upon record creation.https://community.opengroup.org/osdu/platform/system/storage/-/issues/161Storage should rollback ingestion when publishing event fails2023-03-09T18:18:37ZThiago SenadorStorage should rollback ingestion when publishing event failsWhen storage service succeeds in saving new records but fails in publishing the event we create an inconsistency in the system since the data are kept in storage but are not notified to search/indexer. In other words, we need to rollback...When storage service succeeds in saving new records but fails in publishing the event we create an inconsistency in the system since the data are kept in storage but are not notified to search/indexer. In other words, we need to rollback the write to storage in case of a failed publish. The fix is trivial: move [these blocks](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/service/PersistenceServiceImpl.java#L92) to [this block](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/service/PersistenceServiceImpl.java#L104).https://community.opengroup.org/osdu/platform/system/storage/-/issues/160ADR - Clean OpenAPI 3.0 Documentation using 'Code First Approach'2023-07-10T08:02:52ZOm Prakash GuptaADR - Clean OpenAPI 3.0 Documentation using 'Code First Approach'## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
While adopting **OpenAPI 3.0** standards using `springdoc`, we end up adding lot of documentation to native controller of each AP...## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
While adopting **OpenAPI 3.0** standards using `springdoc`, we end up adding lot of documentation to native controller of each API.
- API contract is not clearly visible
- reduces the readability of the API
- business logic & documentation at the same place
## Tradeoff Analysis
- To maintain clean API documentation
- API, Controller segregation
- adopt future changes w.r.t to documentation or contract change
## Proposed Solution:
- Introduce API, Controller Layer Segregation
- API will have contract, definitions & OpenAPI documentation
- Controller will implement the API contract with clean code
#References:
1. [‘Code First’ API Documentation](https://reflectoring.io/spring-boot-springdoc/)
## Sample Refactor in Storage Patch API
- [Patch API](https://community.opengroup.org/osdu/platform/system/storage/-/blob/az/td-codefirst/storage-core/src/main/java/org/opengroup/osdu/storage/api/PatchApi.java)
- [Patch Controller](https://community.opengroup.org/osdu/platform/system/storage/-/blob/az/td-codefirst/storage-core/src/main/java/org/opengroup/osdu/storage/api/PatchController.java)
## Sample Example code
Lets consider a TODO API with normal Crud operation
First we write Interface and define necessary annotations.
```
@RequestMapping("/api/todos")
@Tag(name = "Todo API", description = "euismod in pellentesque ...")
interface TodoApi {
@GetMapping
@ResponseStatus(code = HttpStatus.OK)
List<Todo> findAll();
@GetMapping("/{id}")
@ResponseStatus(code = HttpStatus.OK)
Todo findById(@PathVariable String id);
@PostMapping
@ResponseStatus(code = HttpStatus.CREATED)
Todo save(@RequestBody Todo todo);
@PutMapping("/{id}")
@ResponseStatus(code = HttpStatus.OK)
Todo update(@PathVariable String id, @RequestBody Todo todo);
@DeleteMapping("/{id}")
@ResponseStatus(code = HttpStatus.NO_CONTENT)
void delete(@PathVariable String id);
}
```
##
Then we derive existing controllers from interface for controller implementation
```
@RestController
class TodoController implements TodoApi {
// method implementations
}
```
## Consequences
- Requires changes across services and code refactoring.
- No Breaking functional changes.M17 - Release 0.20Chad LeongOm Prakash GuptaChad Leonghttps://community.opengroup.org/osdu/platform/system/storage/-/issues/159Storage adds null meta to record ingested without2023-03-22T04:11:53ZAn NgoStorage adds null meta to record ingested without1. Record was ingested without specifying "meta" block. PUT api was successful.
2. Fetch the ingested record. Notice that Storage added "meta": null to the record.
**Checking with Search.**
Search indexed successfully. Status code was 2...1. Record was ingested without specifying "meta" block. PUT api was successful.
2. Fetch the ingested record. Notice that Storage added "meta": null to the record.
**Checking with Search.**
Search indexed successfully. Status code was 200.
Search result does not return the meta.
The current behavior is challenged saying that Meta block shouldn't have been added. Or if added, then it should be empty and not null.
So instead of adding:
"meta": null
It should be:
"meta": []
Upon creating or updating a record, providing an empty meta block should also be allowed.https://community.opengroup.org/osdu/platform/system/storage/-/issues/156ADR: Recover a soft deleted record in storage2023-09-11T08:27:45ZAbhishek NandaADR: Recover a soft deleted record in storageAbility to recover a soft deleted record in storage service
# Decision Title
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The storage service provides 2 ways to delete a r...Ability to recover a soft deleted record in storage service
# Decision Title
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
The storage service provides 2 ways to delete a record. One way is to logically delete the record in which the record with same id can be revived later because its version history is maintained and the other one is to purge the record in which case, the record's version history is deleted too. In both types of deletions, the record cannot be accessed using storage or search service.
Today there is no easy way to query or recover the soft-deleted records. Providing admin-only APIs will help admins to search, view and recover the soft-deleted data if required.
# Tradeoff Analysis - Input to decision
Today users have to maintain the soft deleted record IDs on their own. Below is the workaround available today to attempt recovery of such records
1. Recreate the record with existing id and random/empty data and meta blocks. This will mark the record as active.
2. Fetch all versions of the record.
3. Fetch the latest version prior to the one just created to get back the actual record data and meta blocks.
4. Recreate the record using the response to create a new version of the record with the appropriate data.
## Decision
Create 3 new APIs as below
1. Fetch deleted records (accessible to _users.datalake.admins_) -> This will fetch a list of records. Since the list can be very long it should return a maximum of 100 records and support a from and to deletion dates filter along with pagination.
![image](/uploads/ca34cf94f3184fba05d2ade6bb502a90/image.png)
2. Recover deleted records by id (accessible to _users.datalake.admins_) -> This will take a list of record ids (max 500) that are to be recovered and return the list of record ids that succeeded as well as failed.
![image](/uploads/ae448c5fb9ed5803101aeba51a4fd7b4/image.png)
3. Recover deleted records by metadata filters (Currently support for only fromDeletedDate and toDeletedDate) (accessible to _users.datalake.admins_) -> This will take filter criteria of records that are to be recovered and return the list of record ids that succeeded as well as failed.
![image](/uploads/2b1d373eed8513e166fba784be4b3250/image.png)
## Consequences
1. This will help users to bulk recover deleted records in a single go.
2. The APIs will help prevent having garbage record versions that had to be created just to make the record active.
3. This will help users to fetch a list of soft deleted records which was not possible earlier.
Open API spec for the service
[storage-recover-swagger.yaml](/uploads/396cc62881dfe5f075f0e987f0313472/storage-recover-swagger.yaml)https://community.opengroup.org/osdu/platform/system/storage/-/issues/154Storage service stale in-memory cache leads to inconsistency.2023-02-15T18:37:33ZNikhil Singh[MicroSoft]Storage service stale in-memory cache leads to inconsistency.We recently uncovered a bug in storage service due to local cache getting stale. The flow can be understood by the following steps.
1. Deletion of a legal tag via legal service delete API --> response 204 No content after successful del...We recently uncovered a bug in storage service due to local cache getting stale. The flow can be understood by the following steps.
1. Deletion of a legal tag via legal service delete API --> response 204 No content after successful deletion
2. Storage service API call made at https://**********/api/storage/v2/push-handlers/legaltag-changed?token=*** --> Goes to a pod P1 of storage service --> Updates the records compliance for all the record associated with the deleted tag in step 1---> Removes the deleted tag from local cache of pod P1.
3. Storage PUT call to create a record with the deleted legal tag--> goes to a pod P2 of storage--> the cache still has that legal tag-->returns 201 created.
At step 3, all calls going to pod p1 returns "Invalid legal tag" but API calls landing on other pods successfully create these records.
The service ITs are failing in transient manner due to this issue.M17 - Release 0.20Nikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/storage/-/issues/150PersistedCollection cannot scale to large values, an upper limit for records ...2022-11-08T09:28:01ZGary MurphyPersistedCollection cannot scale to large values, an upper limit for records is neededPersisted Collections have been seen lately in various environments that are getting somewhat pathological, meaning they are straining the limits of what the consuming services (mainly Storage and Search) can handle. As the number of it...Persisted Collections have been seen lately in various environments that are getting somewhat pathological, meaning they are straining the limits of what the consuming services (mainly Storage and Search) can handle. As the number of items in a Persisted Collection rise, they will increase the size of the Storage record beyond practical limits as well as put a heavy load on Indexing and Search as they are updated.
An exact limit is a bit tricky to specify, but experience with 100K records has shown increased 500 return codes from Storage and Search when counts are in that neighborhood (100K).
Based on the above behavior (and the upcoming introduction of Collaboration Spaces which provide a scalable solution with transactions and promotion capabilities), it is proposed to introduce a practical limit for sizes of Persisted Collections. A straw man number could be on the order of 50K records mentioned in the Persisted Collection. Counts higher than that would trigger an error on Storage PUT and meaningful response text.
Collaboration Spaces will hopefully be the correct home for controlled collections of massive size (1M records is considered reasonable) since updates can be done via distributed transactions and no single Storage record has to scale to contain the contents of the collection. In the meantime, a limit for Persisted Collections is needed.
[Collaboration Spaces](https://gitlab.opengroup.org/osdu/subcommittees/ea/work-products/adr-elaboration/-/issues/48)
[PersistedCollection](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/work-product-component/PersistedCollection.1.0.0.md)https://community.opengroup.org/osdu/platform/system/storage/-/issues/146POST /query/records:batch with normalization stops converting after 1 convers...2022-10-28T08:04:37ZAn NgoPOST /query/records:batch with normalization stops converting after 1 conversion failureAn attribute was defined as a number in the schema:
```
"depthA": {
"title": "depthA",
"type": "number"
}
```
The meta specified is to convert the values in depthA from ft to meter.
```
"meta": [
{
"...An attribute was defined as a number in the schema:
```
"depthA": {
"title": "depthA",
"type": "number"
}
```
The meta specified is to convert the values in depthA from ft to meter.
```
"meta": [
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
"propertyNames": [
"depthA",
"depthB"
],
```
The record was ingested/created with an empty string assigned to depthA.
```
"data": {
"depthA": "",
"depthB": 123,
"depthC": 456
},
```
Upon record creation, fetch API was called to normalize the record before indexing.
The conversion failed to convert depthA. An error was logged. Fetch API returned a 200, but with a conversion error.
![image](/uploads/28575874041594004a487f3ee009f1f9/image.png)
After this error, the API skipped conversion for other attributes.
Indexer saw this error and returned a 400 status. Trace index trace returns:
```
"statusCode": 400,
"trace": [
"Unit conversion: illegal value for property depthA"
]
```
**Action:** API should continue to convert all specified attributes, and log the conversion errors for those that failed.