Storage issueshttps://community.opengroup.org/osdu/platform/system/storage/-/issues2024-03-15T13:22:20Zhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/219Records created with special characters are not discoverable2024-03-15T13:22:20ZAbhishek Kumar (SLB)Records created with special characters are not discoverableStorage service allows user to a create record with encoded special character.
However, if we try to get the created record storage service return 404.
**Actual ID:** winter-aker-bp-super-sprint-5:reference-data--UnitOfMeasure:m/h
<br>
...Storage service allows user to a create record with encoded special character.
However, if we try to get the created record storage service return 404.
**Actual ID:** winter-aker-bp-super-sprint-5:reference-data--UnitOfMeasure:m/h
<br>
**Encoded ID**: winter-aker-bp-super-sprint-5:reference-data--UnitOfMeasure:m%2fh
The Storage POST endpoint allows user to create storage records with encoded ids:
![image](/uploads/6a923c3582dcb993eaf8d84e2ff32166/image.png)
But the problem arises when user tries to retrieve the record using get endpoint:
`{
"code": 400,
"reason": "Validation error.",
"message": "{\"errors\":[\"Not a valid record id. Found: winter-aker-bp-super-sprint-5:reference-data--UnitOfMeasure:m%2fh\"]}"
}`
The same record do appears in the search result:
![image](/uploads/566be3486c02de5956b2c96e10709d2e/image.png)Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/storage/-/issues/217ADR: Delete record versions2024-03-26T09:49:14ZNeelesh ThakurADR: Delete record versions<a name="TOC"></a>
[[_TOC_]]
# Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
# Problem Statement
Storage service allows creation of record versions without any restrictions. Once number of reco...<a name="TOC"></a>
[[_TOC_]]
# Status
- [x] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
# Problem Statement
Storage service allows creation of record versions without any restrictions. Once number of record versions goes beyond certain limit (e.g. 1K for Azure), it's very costly to fetch the record. Please refer to [ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/178) for more details.
[ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/178) implementation restricts the maximum number of versions to avoid any accidently cost spikes.
This ADR proposes a solution to delete record versions.
[Back to TOC](#TOC)
# Proposed solution
Storage API should provide a new endpoint to delete record versions. It will permanently delete record versions and operation cannot be undone. Users must be member of 'users.datalake.admins' role and OWNER of the record
<details>
<summary>API specification</summary>
```yaml
"/records/{id}/versions":
delete:
tags:
- Records
summary: Purge record versions
description: "The API performs the permanent physical deletion of the given record versions excluding latest version and any linked records or files if there are any.
If 'from' query parameter is used then it will delete all versions before current one (exclusive). If 'limit' query parameter is used then it will delete oldest versions defined by 'limit'.
If both 'from' and 'limit' are used then API will delete 'limit' number of versions starting 'from' version. Instead of using 'limit' or 'from', list of versions can be provided on 'versionIds' query parameter.
API will delete all versions defined by 'versionIds' query parameter. Maximum 50 record versions can be deleted per request.
This operation cannot be undone. Required roles: 'users.datalake.admins' who is the OWNER of the record."
operationId: Purge record versions
parameters:
- name: id
in: path
description: Valid record id following "^[\\w\\-\\.]+:[\\w-\\.]+:[\\w\\-\\.\\:\\%]+$" pattern
required: true
schema:
type: string
- name: from
in: query
description: Record version id from which all record versions aside from the current one are deleted
required: false
schema:
type: long
- name: limit
in: query
description: Number of oldest record versions to be deleted. Value must not exceed number of record versions (excluding latest version)
required: false
schema:
type: integer
- name: versionIds
in: query
description: Comma separated version list (excluding latest version) to be deleted. Maximum 50 versions can be deleted per request.
required: false
schema:
type: integer
- $ref: "#/components/parameters/data-partition-id"
responses:
"204":
description: Record deleted successfully.
"400":
description: Validation error.
content:
application/json:
schema:
$ref: "#/components/schemas/AppError"
"403":
description: User not authorized to perform the action.
content:
application/json:
schema:
$ref: "#/components/schemas/AppError"
"404":
description: Record not found.
content:
application/json:
schema:
$ref: "#/components/schemas/AppError"
"500":
description: Unknown Error.
content:
application/json:
schema:
$ref: "#/components/schemas/AppError"
security:
- bearer: []
```
</details>
[Back to TOC](#TOC)
# Consequences
- New API added to Storage service
- New endpoint is available on Storage service's swagger page
- Tutorial is updated with new endpoint
[Back to TOC](#TOC)M23 - Release 0.26Neelesh ThakurNeelesh Thakurhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/215Increase timeout for storage service requests2024-02-01T12:46:24ZSudesh TagadpallewarIncrease timeout for storage service requestsWhen registering dataset using `/registerDataset` some users are getting 400 error. As per the Logs this request is timing out(with the error- **Unexpected error sending to URL http://storage/api/storage/v2/records METHOD PUT error java....When registering dataset using `/registerDataset` some users are getting 400 error. As per the Logs this request is timing out(with the error- **Unexpected error sending to URL http://storage/api/storage/v2/records METHOD PUT error java.net.SocketTimeoutException: Read timed out**) when it tries to upsertRecord in the Storage.
We have found out that when dataset service is calling storage service and it is taking more than 5 seconds which results in a SocketTimeoutException.
When creating `StorageService` instance using `StorageFactory`, new `HttpClient()` instance is used which has default timeout of 5 seconds. Instead of using new `HttpClient` instance `HttpClientHandler` instance should have been used which has 60 seconds timeout. This code is present in the core-common library. See attached image for reference. ![storage](/uploads/5d81a52c9a968975ad40a538088a57dc/storage.JPG)https://community.opengroup.org/osdu/platform/system/storage/-/issues/214CRS Conversion not working if persistableReferenceCrs left off --> Move Stora...2024-03-08T00:30:17ZBryan DawsonCRS Conversion not working if persistableReferenceCrs left off --> Move Storage to crs-conversion v3According to the documentation we are supposed to be able to leave off the `persistableReferenceCrs` now (although, because of it being required in the schema the best we can do is make it an empty string) and that the CRS conversions wi...According to the documentation we are supposed to be able to leave off the `persistableReferenceCrs` now (although, because of it being required in the schema the best we can do is make it an empty string) and that the CRS conversions will use the `CoordinateReferenceSystemID` to look up the value of the persistable reference. However, this does not appear to work.
Take the following simple Well for example:
```json
{
"acl": {
"owners": [
"data.default.owners@dp.myosdu.com"
],
"viewers": [
"data.default.viewers@dp.myosdu.com"
]
},
"data": {
"FacilityName": "Dummy 1 - Do Not Use Me",
"SpatialLocation": {
"AsIngestedCoordinates": {
"CoordinateReferenceSystemID": "dp:reference-data--CoordinateReferenceSystem:Geographic2D:EPSG::4326:",
"features": [
{
"geometry": {
"coordinates": [
-45.944904,
18.12565
],
"type": "AnyCrsPoint"
},
"properties": {},
"type": "AnyCrsFeature"
}
],
"persistableReferenceCrs": "",
"type": "AnyCrsFeatureCollection"
},
"SpatialGeometryTypeID": "dp:reference-data--SpatialGeometryType:Point:"
}
},
"id": "dp:master-data--Well:TEST_CRS_METHODS",
"kind": "osdu:wks:master-data--Well:1.3.0",
"legal": {
"legaltags": [
"dp-default-legal"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [ ]
}
```
Which should populate the Wgs84Coordinates when sent to the indexer, but you do not see it in the indexer record:
```json
{
"acl": {
"owners": [
"data.default.owners@dp.myosdu.com"
],
"viewers": [
"data.default.viewers@dp.myosdu.com"
]
},
"authority": "osdu",
"createTime": "2024-01-23T19:16:46.001Z",
"createUser": "bryan.j.dawson@exxonmobil.com",
"data": {
"FacilityName": "Dummy 1 - Do Not Use Me",
"SpatialLocation.SpatialGeometryTypeID": "dp:reference-data--SpatialGeometryType:Point:",
"VirtualProperties.DefaultLocation.SpatialGeometryTypeID": "dp:reference-data--SpatialGeometryType:Point:",
"VirtualProperties.DefaultName": "Dummy 1 - Do Not Use Me"
},
"id": "dp:master-data--Well:TEST_CRS_METHODS",
"kind": "osdu:wks:master-data--Well:1.3.0",
"legal": {
"legaltags": [
"dp-default-legal"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"namespace": "osdu:wks",
"source": "wks",
"tags": {
"normalizedKind": "osdu:wks:master-data--Well:1"
},
"type": "master-data--Well",
"version": 1706037405365334
}
```
However, if we fill in the `persistableReferenceCrs` and meta block it works as expected.
```json
{
"acl": {
"owners": [
"data.default.owners@dp.myosdu.com"
],
"viewers": [
"data.default.viewers@dp.myosdu.com"
]
},
"data": {
"FacilityName": "Dummy 1 - Do Not Use Me",
"SpatialLocation": {
"AsIngestedCoordinates": {
"CoordinateReferenceSystemID": "dp:reference-data--CoordinateReferenceSystem:Geographic2D:EPSG::4326:",
"features": [
{
"geometry": {
"coordinates": [
-45.944904,
18.12565
],
"type": "AnyCrsPoint"
},
"properties": {},
"type": "AnyCrsFeature"
}
],
"persistableReferenceCrs": "{\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4326\"},\"name\":\"GCS_WGS_1984\",\"type\":\"LBC\",\"ver\":\"PE_10_9_1\",\"wkt\":\"GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4326]]\"}",
"type": "AnyCrsFeatureCollection"
},
"SpatialGeometryTypeID": "dp:reference-data--SpatialGeometryType:Point:"
}
},
"id": "dp:master-data--Well:TEST_CRS_METHODS",
"kind": "osdu:wks:master-data--Well:1.3.0",
"legal": {
"legaltags": [
"dp-default-legal"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [
{
"coordinateReferenceSystemID": "dp:reference-data--CoordinateReferenceSystem:Geographic2D:EPSG::4326:",
"kind": "CRS",
"name": "WGS 84",
"persistableReference": "{\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4326\"},\"name\":\"GCS_WGS_1984\",\"type\":\"LBC\",\"ver\":\"PE_10_9_1\",\"wkt\":\"GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4326]]\"}",
"propertyNames": [
"SpatialLocation.AsIngestedCoordinates"
]
}
]
}
```
and indexer record looks like what we expected
```json
{
"acl": {
"owners": [
"data.default.owners@dp.myosdu.com"
],
"viewers": [
"data.default.viewers@dp.myosdu.com"
]
},
"authority": "osdu",
"createTime": "2024-01-23T19:16:46.001Z",
"createUser": "bryan.j.dawson@exxonmobil.com",
"data": {
"FacilityName": "Dummy 1 - Do Not Use Me",
"SpatialLocation.SpatialGeometryTypeID": "dp:reference-data--SpatialGeometryType:Point:",
"SpatialLocation.Wgs84Coordinates": {
"geometries": [
{
"coordinates": [
-45.944904,
18.12565
],
"type": "point"
}
],
"type": "geometrycollection"
},
"VirtualProperties.DefaultLocation.IsDecimated": false,
"VirtualProperties.DefaultLocation.SpatialGeometryTypeID": "dp:reference-data--SpatialGeometryType:Point:",
"VirtualProperties.DefaultLocation.Wgs84Coordinates": {
"geometries": [
{
"coordinates": [
-45.944904,
18.12565
],
"type": "point"
}
],
"type": "geometrycollection"
},
"VirtualProperties.DefaultName": "Dummy 1 - Do Not Use Me"
},
"id": "dp:master-data--Well:TEST_CRS_METHODS",
"kind": "osdu:wks:master-data--Well:1.3.0",
"legal": {
"legaltags": [
"dp-default-legal"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"modifyTime": "2024-01-23T19:33:06.905Z",
"modifyUser": "bryan.j.dawson@exxonmobil.com",
"namespace": "osdu:wks",
"source": "wks",
"tags": {
"normalizedKind": "osdu:wks:master-data--Well:1"
},
"type": "master-data--Well",
"version": 1706038386690196
}
```
:pushpin: **Update from @nthakur :**
> `CoordinateReferenceSystemID` dynamic lookup feature was implemented on crs-conversion `v3` endpoint.
> Storage service conversion endpoint (used by indexer-service) calls crs-conversion service to get converted records. As of current release (M22), Storage service is using `v2` endpoints from crs-conversion. This is the reason we see this behavior.
So to solve this one will require moving to the v3 endpoints for crs-conversion service.https://community.opengroup.org/osdu/platform/system/storage/-/issues/213Discrepancy in Storage API for create/update record operation2024-01-31T23:00:31ZNeha KhandelwalDiscrepancy in Storage API for create/update record operationFor Storage create/update record API, if a record ID ends in a dot (.) the data block for the record is not properly uploaded to the Microsoft storage account. In cases, where the records for create/update multiple record operation have ...For Storage create/update record API, if a record ID ends in a dot (.) the data block for the record is not properly uploaded to the Microsoft storage account. In cases, where the records for create/update multiple record operation have the similar IDs only differentiated by a dot at the end (ex. M/M.), the data block will be the same for both records. The issue is that Microsoft storage accounts do not support directory names ending with a dot (.), a forward slash (/), or a backslash (\\) and path segments ending with a dot ([https://learn.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata](https://learn.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata "https://learn.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata")). When uploading the block blob with the record data to the storage container, the BlobStore.class tries to use a path with the record ID as a folder, such as
\<kind\>/\<partition\>:reference-data--ExternalUnitOfMeasure:LIS-LAS::**M.**/1704916580557751
but \<partition\>:reference-data--ExternalUnitOfMeasure:LIS-LAS::**M.** is not a valid directory name so the dot at the end is ignored, and block blob is uploaded to \<partition\>:reference-data--ExternalUnitOfMeasure:LIS-LAS::**M** instead. This was also manually confirmed by trying to upload a blob to a folder with a name ending in dot.
It is a corner case but this issue has impacted RDD values for M and M. on all partitions
* For example on "prod-weu-des-prod-testing-eu", the records prod-weu-des-prod-testing-eu:reference-data--ExternalUnitOfMeasure:LIS-LAS::M. and prod-weu-des-prod-testing-eu:reference-data--ExternalUnitOfMeasure:LIS-LAS::M has same "M." values in ID, Code and Symbol field. These were created using RDD script/pipeline.
* Impact is on all partitions for \<partition\>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M. and \<partition\>:reference-data--ExternalUnitOfMeasure:LIS-LAS::M values
Proposed solution is to reject record IDs that end in these unsupported characters (i.e. return 400 bad request when such record IDs are used).https://community.opengroup.org/osdu/platform/system/storage/-/issues/212GeoJson validation2024-03-15T14:11:20ZAdam ChengGeoJson validationThis is a linked issue between the Storage API and Search API.
When I ingest an new object witha invalid GeoJSON (e.g. polygon is is not close). It will pass the Storage API as it mainly check types. But it will silently failed indexing...This is a linked issue between the Storage API and Search API.
When I ingest an new object witha invalid GeoJSON (e.g. polygon is is not close). It will pass the Storage API as it mainly check types. But it will silently failed indexing and never show up on Search API.
A related issue: currently it take up to 30 seconds before a newly ingested object shows up on the Search API. It makes a bit challenging for a near real-time application.
Possible solution:
An additional query param on the PUT `/records` endpoint. If the param is set, the operation will only be successful when it finished indexing.
It would be ideal for ingestion and indexing/discovery operations to be atomichttps://community.opengroup.org/osdu/platform/system/storage/-/issues/211Different behavior on delete endpoint2024-03-15T14:13:21ZAdam ChengDifferent behavior on delete endpointThere are currently two endpoints for deleting an object>
1. Deleting multiple objects: /records/delete
2. Deleting single object /records/<Object_id>:delete
When attempt to delete an object that is already been deleted:
Endpoint 1 will...There are currently two endpoints for deleting an object>
1. Deleting multiple objects: /records/delete
2. Deleting single object /records/<Object_id>:delete
When attempt to delete an object that is already been deleted:
Endpoint 1 will return 204 while endpoint 2 will return 404
It would be desirable if endpoint 1 returns some sort of error if one of/multiple objects has already been deleted or non-existencehttps://community.opengroup.org/osdu/platform/system/storage/-/issues/195Unhandled Exceptions for missing required attributes while creating record2024-03-15T14:13:45ZAnubhav BajajUnhandled Exceptions for missing required attributes while creating recordIssue Currently, the storage PUT endpoint lacks proper error messages if there are missing attributes of in the payload. It shows a generic message which is not informative enough for user to address it: "HV000028: Unexpected exception d...Issue Currently, the storage PUT endpoint lacks proper error messages if there are missing attributes of in the payload. It shows a generic message which is not informative enough for user to address it: "HV000028: Unexpected exception during isValid call,"
Ideally the error message should clearly list out the missing attributes such as 'kind', 'acl', or 'legal'.
Below sample example where acl in null , Response gives generic message.
![image](/uploads/738760ea17a8bce24fc4615d5d26920b/image.png)
Suggestions
• Add cases where these required attributes are null. With relevant error messages like
| Missing Attributes | Suggested Error Messages |
|--------------------|--------------------------|
| Kind | Mandatory fields missing- kind / kind cannot be empty |
| Acl | Mandatory fields missing- acl / acl cannot be empty |
| Legal | Mandatory fields missing- legal / legal cannot be empty |
| Acl and Legal | Mandatory fields missing- acl, Mandatory fields missing- legal / acl cannot be empty, legal cannot be empty |
| Kind, Acl and Legal | Mandatory fields missing- kind, Mandatory fields missing- acl, Mandatory fields missing- legal / kind cannot be empty, acl cannot be empty, legal cannot be empty |https://community.opengroup.org/osdu/platform/system/storage/-/issues/193How to troubleshoot? Field missed from Search response although we see it fro...2023-12-16T16:45:21ZDebasis ChatterjeeHow to troubleshoot? Field missed from Search response although we see it from Storage response.Companion issue in Preship site is here -
https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/649
This is not a case of anything linked to conversion (using Meta block) or typo error in the field name.
The real question...Companion issue in Preship site is here -
https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/649
This is not a case of anything linked to conversion (using Meta block) or typo error in the field name.
The real question is - how to troubleshoot this kind of problem?
cc @nthakur and @gehrmannhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/192RAFSDDMS Unit conversion issue2023-12-05T13:46:05ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRAFSDDMS Unit conversion issueIt was observed that the record from the collection: https://community.opengroup.org/osdu/qa/-/tree/main/Dev/48_CICD_Setup_RAFSDDMSAPI?ref_type=heads
Requested with conversion headers:
```plaintext
curl --location 'https://community.g...It was observed that the record from the collection: https://community.opengroup.org/osdu/qa/-/tree/main/Dev/48_CICD_Setup_RAFSDDMSAPI?ref_type=heads
Requested with conversion headers:
```plaintext
curl --location 'https://community.gcp.gnrg-osdu.projects.epam.com/api/storage/v2/query/records:batch' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: osdu' \
--header 'accept: application/json' \
--header 'frame-of-reference: units=SI;crs=wgs84;elevation=msl;azimuth=true north;dates=utc;' \
--header 'Authorization: Bearer ' \
--data '{
"records": [
"osdu:work-product-component--RockSampleAnalysis:Test"
]
}'
```
Causing internal server error:
```plaintext
Caused by: java.lang.NullPointerException: Cannot invoke "com.google.gson.JsonArray.size()" because "elementArray" is null
at org.opengroup.osdu.core.common.util.JsonUtils.overrideNestedNumberPropertyOfJsonObject(JsonUtils.java:219)
at org.opengroup.osdu.core.common.util.JsonUtils.overrideNumberPropertyOfJsonObject(JsonUtils.java:146)
at org.opengroup.osdu.core.common.crs.UnitConversionImpl.convertRecordToSIUnits(UnitConversionImpl.java:166)
at org.opengroup.osdu.core.common.crs.UnitConversionImpl.convertUnitsToSI(UnitConversionImpl.java:56)
at org.opengroup.osdu.storage.conversion.DpsConversionService.doConversion(DpsConversionService.java:80)
at org.opengroup.osdu.storage.service.BatchServiceImpl.fetchMultipleRecords(BatchServiceImpl.java:228)
at org.opengroup.osdu.storage.api.QueryApi.fetchRecords(QueryApi.java:135)
```
Further investigation is required to fix it.https://community.opengroup.org/osdu/platform/system/storage/-/issues/191Add /liveness_check2024-01-08T10:07:15ZRiabokon Stanislav(EPAM)[GCP]Add /liveness_checkNeed to add the endpoint '/liveness_check' to verify the operational status of the Storage Service.Need to add the endpoint '/liveness_check' to verify the operational status of the Storage Service.M23 - Release 0.26Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/storage/-/issues/190ADR Consumer Topic Identification [ Replay Design ]2024-02-21T13:15:35ZAkshat JoshiADR Consumer Topic Identification [ Replay Design ]<h2>ADR Consumer Topic Identification</h2>
## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
<h3>Problem Context</h3>
Today, the storage service publishes **RecordChange messages to “recordstopi...<h2>ADR Consumer Topic Identification</h2>
## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
<h3>Problem Context</h3>
Today, the storage service publishes **RecordChange messages to “recordstopic”.**
When Storage Service publishes a **RecordChange** message to the “**recordstopic**” topic of the service bus, all the consumers get notified (eg. Indexer service, notification service). During scenarios like replaying for **reindex** scenarios, notifying all the consumers may not be required. Hence, we need a way to instruct storage service to publish **RecordChange** messages to a custom topic depending on the use case. For example, if the replay is going to be done for re-index, then we can instruct storage service to publish the **RecordChange** messages to a “reindex” topic which is being listened to by the indexer only, instead of publishing them to **recordstopic which has many consumers.** This will ensure that only the indexer service gets notified of the events.
Therefore, it is of utmost importance that the Producer's design allows for the appropriate routing of operations to their intended topics. This brings us to the question of how the Storage service can accurately determine the topic to which each message should be directed based on its specific functionality/operation . In response to this challenge, we have explored the following design, which will serve as the foundation for the development of our Replay API.
<table>
<tr>
<td><strong> Design Option </strong>
</td>
<td><strong> Detailed Approach </strong>
</td>
<td><strong> Pros/Cons</strong>
</td>
</tr>
<tr>
<td>
<ol>
<ul>1. <strong> Create different Topic for Each Operation and provide operation name i.e. reindex as input to the replay API</strong>.
<p><span style="color: green;"> [Preferred Approach] </span></p>
<ul>
</ol>
</td>
<td>There will be a separate topic for each operation.
<p>
For example, indexer service can listen to a topic called “reindex” and notification service can listen to the topic “notify” in addition to “recordstopic”.
<p>
The replay API will take the input as operation name i.e. reindex, based on that, it will decide which topic the replay API has to publish the recordChange message. This will ensure only the indexer gets notified.
<p>
![Picture1](/uploads/37e29ff02caff81f2f62ff3e98cabb74/Picture1.png)
<p>
<strong>Note</strong> – One operation will maps to one Topic in Service ( 1:1) . While a single topic can have multiple consumers.
</td>
<td><strong> Pros: </strong>
<ul>
<li>Abstraction and statelessness as users need not know about internal topics.
<li>Consistency as different CSP can decides on common operation name irrespective of internal implementation details.
<li>Decoupling of the internal implementation from Replay operation.
<p>
<strong>Cons: </strong>
<ul>
<li>Management of mapper which helps us to map the functionality i.e. reindex to topic name.
<li>Implementation will take time.
<li>Producers should know about consumer topic mapping. [ <strong>Remark</strong> – every Producer knows topic names either through registry or in memory store or environment variable mapper, currently we pass it as hardcoded value from deployment yaml to application properties]
</li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<td>
<ol>
<ul>2. <strong>Create different Topic for Each Operation and provide Topic Name/ID as input to the replay API</strong>.
</ul>
</ol>
</td>
<td>There will be a separate topic for each operation.
<p>
For example, indexer service can listen to a topic called “reindex” and notification service can listen to the topic “notify” in addition to “recordstopic”.
<p>
If Replay is required for reindex scenario, then replay API can be called with the parameter topicId’s value as “reindex”.
<p>
This will cause storage services to publish recordChange messages to reindex topic instead of recordchangetopic. This will ensure only the indexer gets notified.
</td>
<td><strong>Pros - </strong>
<ul>
<li> No need to maintain the internal mapper if we use topic name.
<li>Implementation will be easy if we use the topic name directly.
<p>
<strong>Cons –</strong>
<ul>
<li> We must keep up with the mapper that allows us to associate topic IDs with their corresponding topic names when utilizing the topic ID as input for the reply to API in case we pass topic id.
<li> Users should have access to the internal topic details in case we use topic name.
<li> Since these APIs will be introduced at the community level, customizing them for specific topics, which may have different names or implementations by different CSPs, could impact uniformity.
</li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<td>
<ol>
<ul>3. <strong> Create Different Topic for Each Consumer and let the specific consumer like indexer, call the replayAPI with the topicId</strong>
</ul>
</ol>
</td>
<td>In this, a new Reindex API in indexer will call the replay API and will provide the topic name along in the request body.
</td>
<td><strong>Pros – </strong>
<ul>
<li>Internal details like topic name need not be known to the user.
<li>The consumer can perform pre-requisite operations like deleting indices before calling the replay API.
<p>
<strong>Cons – </strong>
<ul>
<li> User has to use different APIs which will be bad experience.
<li>If we call it using reindex is that I must change the response payload to incorporate the status id and ADR in community and code merge.
</li>
</ul>
</li>
</ul>
</td>
</tr>
</table>
**Conclusion** - We are going with approach 1 taking into consideration the Pros.Akshat JoshiAkshat Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/issues/189[SAST] Vue_DOM_XSS in file index.html2023-11-15T10:54:25ZYauhen Shaliou [EPAM/GCP][SAST] Vue_DOM_XSS in file index.html**Description**
The method m-1"\> embeds untrusted data in generated output with href, at line 36 of \\storage\\provider\\storage-azure\\src\\main\\resources\\static\\index.html. This untrusted data is embedded into the output without p...**Description**
The method m-1"\> embeds untrusted data in generated output with href, at line 36 of \\storage\\provider\\storage-azure\\src\\main\\resources\\static\\index.html. This untrusted data is embedded into the output without proper sanitization or encoding, enabling an attacker to inject malicious code into the generated web-page.
# **Location:**
<table>
<tr>
<th> </th>
<th>Source</th>
<th>Destination</th>
</tr>
<tr>
<th>File</th>
<td>storage/provider/storage-azure/src/main/resources/static/index.html</td>
<td>storage/provider/storage-azure/src/main/resources/static/index.html</td>
</tr>
<tr>
<th>Line number</th>
<td>92</td>
<td>36</td>
</tr>
<tr>
<th>Object</th>
<td>pathname</td>
<td>href</td>
</tr>
<tr>
<th>Code line</th>
<td>return location.protocol + '//' + location.host + location.pathname</td>
<td>
\<a :href="signInUrl" class="btn btn-primary" v-if="!token" class="col-2"\>Login\</a\>
</td>
</tr>
</table>M21 - Release 0.24https://community.opengroup.org/osdu/platform/system/storage/-/issues/187ADR: Replay API2024-02-29T14:40:15ZAkshat JoshiADR: Replay APITwo New API in Storage service as part of Replay flow will be introduce in the storage service.
* [] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
## Context & Scope
This ADR is centered around the design of ...Two New API in Storage service as part of Replay flow will be introduce in the storage service.
* [] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
## Context & Scope
This ADR is centered around the design of the new replay API within OSDU's storage service which is introduced as the part of the [Replay ADR](https://community.opengroup.org/osdu/platform/system/storage/-/issues/186). The purpose of this Replay API is to publish messages that indicate changes to records, which are subsequently received and processed by consumers. It's important to note that the handling of these messages follows an idempotent process.
## Terminology
<table>
<tr>
<td><strong> Name</strong>
</td>
<td><strong> Explanation</strong>
</td>
</tr>
<tr>
<td><strong> Record</strong>
</td>
<td>The record is stored in OSDU Data Platform in two parts, i.e., document database, which contains basic data (id, kind, legal information, and access permissions), and file storage in a Java Script Object Notation (JSON) format, which contains other relevant information of the record. We are interested in the document database part.
</td>
</tr>
</table>
## Tradeoff Analysis
The new APIs does not represent a breaking change of any other API, and consequently neither for the consuming applications. Only concerned-consuming applications would benefit from this new feature, while it remains entirely transparent for others.
## Additional Requirement
The newly introduced APIs must facilitate [Collaboration workflows](https://community.opengroup.org/osdu/platform/system/storage/-/issues/149) through the utilization of the x-collaboration header. Additionally, the replay mechanism should ensure the accurate publication of collaboration context information in the corresponding event.
## Decision
The proposal is to provide POST and GET Replay API -
The new APIs does not represent a breaking change of any other API, and consequently neither for the consuming applications. Only concerned-consuming applications would benefit from this new feature, while it remains entirely transparent for others.
<table>
<tr>
<td><strong> API fields </strong>
</td>
<td><strong>Explanation</strong>
</td>
</tr>
<tr>
<td><strong>kind</strong>
</td>
<td>It specifies to what Kind the schema belongs to. [optional]
</td>
</tr>
<tr>
<td><strong>repalyId</strong>
</td>
<td>It represents status id. [required]
</td>
</tr>
<tr>
<td><strong>operation</strong>
</td>
<td> Define the replay operation to be carried out. [required]
</td>
</tr>
<tr>
<td><strong>filter</strong>
</td>
<td> Define based on which field the record is selected. [optional]
</td>
</tr>
</table>
<strong>Allowed roles for API access</strong> : users.datalake.ops
<br>
<table>
<tr>
<td>
<strong>Method</strong>
</td>
<td>
<strong> API Endpoint</strong>
</td>
<td>
<strong>Design</strong>
</td>
</tr>
<tr>
<td> POST
</td>
<td>v1/replay
</td>
<td>
<strong>Request Example - </strong>
<p>
<strong> </strong>
<p>
1. <strong>Description</strong> - This API request will reindex all the storage records.
<p>
This phase we will pass empty body for reindexall
<p>
{
<p>
}
<p>
In next phase -
<p>
![operationrepaly](/uploads/d7679bf7d4d6d9745e0d9c579905fc74/operationrepaly.png)
<p>
2. <strong>Description</strong> - This API request will reindex the specific kinds of storage records in this operationName is optional by default, it will reindex specific kinds with filter field. Currently we will replay for single kind only so the array of kind will be restricted to size one.
<p>
![operationrepaly](/uploads/f06805a167d15986688ba23ac85ee897/operationrepaly.png)
<p>
<p>
<strong>Response example – </strong>
![responsepostreplay](/uploads/c557910f6369deda3971866bd2130864/responsepostreplay.png)
<p>
<strong>
</td>
</tr>
<tr>
<td> GET
</td>
<td>
replay/status/{id}</em>
<p>
</td>
<td>
<strong>Request:</strong>
<p>
<p>
<p>
1. <strong>Response Replay in Progress:</strong> <br>
<p>
a) <b>Scenario</b> - In Replay All <br><br>
![replaystatusAllKind](/uploads/12f155b5d491010f3ea37c2576e56e19/replaystatusAllKind.png) <br>
b) <b>Scenario</b> - In Replay single kind <br><br> ![replaystatusforsinglekind](/uploads/2043d80e2d350faa2f3fdb41d4601e0f/replaystatusforsinglekind.png)
<br>
<p>
<p>
2. <strong>Response Replay in Failed:</strong> <br>
<p>
a) <b>Scenario</b> - In Replay All <br><br>
![replayFailedForAllKind](/uploads/3d9a64803b229d3b46d4e283047d285e/replayFailedForAllKind.png)
<br>
b) <b>Scenario</b> - In Replay single kind <br><br>
![replayfailedforsinglekind](/uploads/407b53b19ddfa4545f52e9e88d34fb11/replayfailedforsinglekind.png)
<p>
<p>
</td>
</tr>
</table>
<br>
API spec swagger yaml -[ReplayAPISpecs.yaml](/uploads/f9e8ddd4958bf04f9bc99994ebdc4e41/ReplayAPISpecs.yaml)https://community.opengroup.org/osdu/platform/system/storage/-/issues/186ADR: Replay2024-03-05T10:59:17ZAkshat JoshiADR: Replay<a name="ppadhi"></a>OSDU - Replay and Replay API
# Table of Contents
[Context ](#_toc119676063)
[Problems with Current Reindex All Solution ](#_toc119676075)
[Replay ](#_toc119676076)
[Requirements to address ](#_toc119676077)
[Arc...<a name="ppadhi"></a>OSDU - Replay and Replay API
# Table of Contents
[Context ](#_toc119676063)
[Problems with Current Reindex All Solution ](#_toc119676075)
[Replay ](#_toc119676076)
[Requirements to address ](#_toc119676077)
[Architectural Options ](#_toc119676078)
[Decision ](#_toc119676079)
[Replay API](#_toc119676080)
## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
## <a name="_toc119676063"></a>Context
This ADR is centered around the design of the new replay flow within OSDU's storage service. The purpose of this Replay flow is to publish messages that indicate changes to records, which are subsequently received and processed by consumers. It's important to note that the handling of these messages follows an idempotent process.
The Replay flow will address following-
1. In case of disaster, this replay flow will help us to rebuild the indexes that to RPO.[Out of Scope of ADR]
2. Reindexing the records by publishing the record change messages to consumer Indexer service.
3. Correction of indices after changes to structure of the storage records of a particular kind.
**Replay rate** - It is the rate at which storage publish the record changes message to service bus.
## <a name="_toc119676075"></a>Problems with Current Reindex All Solution
|**Problem**|**Details**|**What is Required?**|
| :- | :- | :- |
|Reliability |<p>**Operation is Synchronous.**</p><p>- Very long HTTP call is never reliable</p><p></p><p></p><p>The Reindex is a synchronous operation, making the operation Unreliable and not resilient to failures. If there is any interruption to the connection, all the status and progress could be lost.</p><p></p>|The operation must be reliable. If the operation is triggered, it must either succeed or it must fail and in both the cases, the user must be diligently informed with the right reasons for success/failures. The system should not be in a state where the user has no clue what’s happening.|
|Resiliency|Abrupt disturbance of the reindex-process leaves the system in an inconsistent state. For example, if there is any exception or if the process crashes, then the system is left entirely in an inconsistent state.|The system must be resilient to failure and must always succeed. If the operation fails, then the system must be left in the previous state.|
|Scale|Due to the synchronous and non-resilient nature of the current implementation, the scale is very limited. It cannot ingest more than a couple of million records reliably.|The reindex operation must scale to any number of records|
|Speed|The speed is very slow. It’s known to take close to an hour for 1 million records.|Faster rate of reindexing is required. For example, 100 million records should not take more than a few hours. |
|Tracking/Observability|There is no way for the user to know about the progress.||
|Pausing/Resuming reindex|Today, there is no capability to pause and resume reindex. Given that this will be a long running operation, having pause and resume will be good to have.||
|No Delta Reindexing|For some Disaster Recovery Scenarios, there may be partial backups available. So reindexing only a subset of records of a kind can prove to be useful. This functionality is not available today.||
|Parallelization|Currently, the reindex is a procedural process. This has impact on both scale as well as speed.||
## <a name="_toc119676077"></a>Requirements to address
To be able to address these issues, we need to re-design the way reindex works, addressing various functional and non-functional aspects like speed, scale, reliability, observability, etc. The below table outlines what is expected out of the new Reindex design.
|**Requirement**|**Details**|**Technical Implications** | **Scope** |
| :- | :- | :- | :- |
|1. Scalability|<p>The Replay operation must be scalable; it should be able to handle infinitely large amounts of records.</p><p><br>A realistic goal to target can be 100m records in 4-5 hours.</p>|<p>Need to ensure Elasticsearch storage can be scaled up.</p><p></p><p>For achieving a higher scale, the following must be done: -</p><p>- The whole operation must be **Asynchronous** in nature</p><p>- It must be resilient to failures due to pod crashes, 429s due to high Database/Service Bus/Elasticsearch load.</p><p>- We can leverage Message Broker to divide and conquer and have the framework.</p><p>- We can also look at job schedulers like QUARTZ to achieve a reliable reindex.</p><p>- Need to evaluate which is the best service to perform this reindexing. </p><p>- Can also try to leverage **Airflow**</p><p></p>| In Scope of ADR |
|2. Reliable Responses|<p>When the operation is triggered, the response must be reliable. </p><p></p><p>There could be some pre-validation done to check whether the reindex process can be completed either successfully or not.</p><p>The result of whether the operation is success or fail, should be communicated via response to the user properly.</p>|Today, we don’t return anything apart from 200 OK in the response even if things fail. <br><br>The entire response should be revamped and reworked on how the status can be conveyed to the user in a useful way.| In Scope of ADR |
|3. Observability and Monitoring|<p>Given the fact that reindex is a long running operation, the User triggering the reindex must have insights into what is going on, using a track status API.</p><p></p><p>Some of the details should include:</p><p>- **Status:** Validating, Stopping-Ingestion, In-progress, Finalizing, Complete, Error, etc.</p><p>- **Progress:** Overall percentage, per index progress, remaining records count, ETA</p><p></p>|We could store the progress in a Redis cache or elsewhere that can be used to report back to the user on the progress.| In Scope of ADR |
|4. Reliable System State – Consistency before/after operation in case of failure|<p>Guarantee to reindex valid storage records – **Must have**</p><p><br>**(depends on message broker reliability)**</p><p></p><p><br>**Rollbacks** – nice to have</p>|<p></p><p>If there are unrecoverable errors during reindexing a particular kind, then that leaves the system in an inconsistent state. It would be good to “**rollback**” the operation to restore the system to the state before the operation was triggered for that kind.</p><p></p><p>There should also be **no concurrent “reindexAll” operation** running. There can however, be concurrent reindex of different kinds happening at the same time.</p><p>It can be a configurable parameter on whether the rollback should be done in case of unrecoverable failures, due to internal system errors.<br><br>How this can be achieved is that, all the reindexed records for a kind, should be indexed into a new “secondary index” for that kind, and only if that is succeeds completely, the index can be renamed and replace the primary index.<br><br>Elasticsearch’s clone index feature can be utilized to achieve this.</p><p></p><p>- Reindex failed record IDs</p>|Out of Scope of ADR |
|5. Stop Ingestion/Search during Reindex|<p>During **Reindex**, the normal ingestion should stop. This is because:</p><p>- There are some edge cases which could end up the system in an inconsistent state. Edge Cases: **<TODO>**</p><p>- Load on Elasticsearch</p><p></p>| | Out of Scope of ADR|
|6. Speed</p>|<p></p><p>The operation is quite slow today. It takes almost an hour to reindex a million records. This means it will take a few days to reindex 100m records, which is not practical.</p><p></p><p>Two Issues:</p><p>1. Finding Unique Kinds</p><p>2. Reindexing – Database load</p>|<p></p><p>This is **directly dependent on the scalability of the underlying infra like Database** and Elasticsearch. </p><p>Database can be scaled up/out on demand, by either the UI by customer (i.e., a via CP), or some other means. </p><p></p><p>Auto scaling-out of Elasticsearch is currently not possible, so we may be limited in speed due to Elasticsearch. We can, however, scale up Elasticsearch and this can help in higher speed.</p><p></p><p>How this scale up is triggered automatically or manually is something we need to evaluate and do a POC.</p><p></p><p>Storage Service’s queries can also be revisited – there was a change done in some service which had a more efficient implementation of paginated queries - [Performance improvement on paginated query for CosmosDB (!244) · Merge requests · Open Subsurface Data Universe Software / Platform / System / Lib / cloud / azure / OS Core Lib Azure · GitLab (opengroup.org)](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/244/diffs)</p><p></p><p></p>| |Out Scope of ADR |
|7. **Delta Reindex** and **Consistency Checker/Enforcer**|<p>Doing a delta reindex can be useful if there is restoration of backups during a disaster recovery. This will result in faster recovery times.</p><p></p><p>Delta Reindex = reindex only those records that are not present in Backup.<br><br>When we talk about delta reindex, we need to ensure there is consistency across all 3 components – storage blob, storage records and Elasticsearch.</p><p></p>|<p>Need to explore feasibility. The operation can be something like Reindex All records whose create/update time > X.</p><p></p><p>A consistency enforcer should be built that will ensure that the 3 entities are in consistent state.</p>| Out Scope of ADR |
|8. Snapshot Backup/Cluster replication|<p>Backup Elasticsearch storage Snapshots frequently, and in case of disaster, restore the snapshot and then perform the delta reindex.<br><br>This will make the recovery times much faster| |Out Scope of ADR |
|9. Source of trigger|During a recovery process, who will make the call to reindex? Is it the user or internal system? |Will need to design and account for this fact in the reindex design.| Out Scope of ADR |
|10. Pause/Resume Reindex|Since reindex is a long running operation, having the ability to pause and resume reindex operation would be nice to have|<p>We need to ensure system consistency when the operation is paused and resumed. </p><p></p><p>Also, any new records ingested after the pause must be included in the reindex process when it’s resumed.</p><p></p>| Out Scope of ADR |
## <a name="_toc119676078"></a> Architectural Options:
<br>
|**Options**|**Pro**|**Cons**|**Work Required**|
| :- | :- | :- | :- |
|1. Using **Airflow** + Message Broker + StorageService + Workflow Service|<p>- Proven Workflow Engine</p><p>- Lesser new implementations in storage services, so lesser work required by other CSPs.</p>|<p>- Process becomes slower and inefficient.</p><p>- Lot of HTTP calls from Airflow <-> AKS</p><p>- Airflow will require access to internal Infrastructure to operate in the most efficient manner.</p><p>- Some required features are not yet available in ADF Airflow </p><p>- Parallelization may spawn up 1000s of tasks waiting to be scheduled. **Scalability can be issue.**</p><p>- Concurrency and Safety guarantee is tricky – allowing no more than one reindex for a kind</p><p></p>|<p>**Airflow**</p><p>- DAG using TaskGroups, Dynamic Task Mapping, Concurrency handling.</p><p>- Build pipelines to integrate new DAG.</p><p></p><p>**Storage Service**</p><p>- Implement new APIs to publish messages to message broker.</p><p></p><p>**Indexer Service**</p><p></p><p>**Workflow Service**</p><p>- Have new APIs to support observability</p><p>- Design for checkpointing</p>|
|2. Using **StorageService** + **Message Broker**|<p>- Simple, Lesser moving parts</p><p>- Fast & Efficient</p>|- Parallelization may require state management.|<p>**Storage Service**</p><p>- New APIs for exposing Replay functionality (ReplayAll, ReplayKind, GetReplayStatus)</p><p>- New Modules for replay message processing</p><p></p><p>**Indexer Service**</p><p>- Delete ALL kinds API</p>|
## <a name="_toc119676079"></a> Decision:
We chose design option 2 using storage service and message broker as the advantage is to persist the replay status and allows to re-play and return the status and simpler implementation.
- **[Decision]** What led us to select the Storage service for the Replay API decision? <br>
* The source of truth for the storage records is – Storage Service. It is the storage service, that publishes the record change messages, which are then consumed by the consumers. This processing of those messages is idempotent.So, it’s fair to say that to trigger reindexing, we must invoke some procedure in storage service, that will make it emit record change messages onto the message broker.<br>
* Indexer is just a consumer of the recordChange messages, and there could be other consumers who require this replay functionality as well. In those cases, instead of letting each consumer build their own replay logic, if we have it in one common place, it would benefit all the consumers. <br>
* This way, one consumer doesn’t have to depend on indexer, which is also just another consumer<br>
* Reindex is just one-use cases, that uses this new Replay functionality. Other consumers can have their own use case for consuming those replayed messages.
<br>
**Design Approach for option 2:**
![Aspose.Words.71972436-70f7-48df-8f1c-d2035f55ce34.004](/uploads/5a573b82493315f91adeee547fd97fee/Aspose.Words.71972436-70f7-48df-8f1c-d2035f55ce34.004.png)
**Note**
The ADR also helps to address following issues - <br>
- **[Issue]** https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/91 <br>
* The Replay flow will include a Service Bus topic for every event. If we need to introduce new events in the future that necessitate message publishing, we can easily do so by introducing a new topic and associated logic. This approach can help prevent unintended consequences that may arise from triggering other listeners on the same topic, as they can be resolved accordingly. <br>
- **[Issue]** https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/66
* Utilizing the service bus and tracking its progress assists us in achieving a reliable design, including the built-in reliability of message queuing. <br>
- **[Issue]** https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/80
* With the flexibility to introduce new topics in the ReindexAkshat JoshiAkshat Joshihttps://community.opengroup.org/osdu/platform/system/storage/-/issues/185ADR: API to retrieve past events of storage records2023-10-11T16:28:52ZYifan YeADR: API to retrieve past events of storage recordsNew API in Storage service to rehydrate past creation and last modified events for a given kind within the given time range.
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
## Context & Scope
The OSDU Sto...New API in Storage service to rehydrate past creation and last modified events for a given kind within the given time range.
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [ ] Approved
* [ ] Retired
## Context & Scope
The OSDU Storage service does not provide a way to retrieve past events of the records been created/modified. Many OSDU applications would be interested in retrieving the past events of the records that happened before the application subscribed to the notification service. This new API proposed in the ADR will provide the concerned applications with a way to backtrack the events.
The proposal is to provide an API on storage service to support retrieving past events of records of a kind that happened in the given time range, where the events will be returned in a paginated format and ascending chronological order based on the timestamp.
The new API will retrieve the first and the last events of the record, filter the events by the start date and end date provided by the user, and then return the filtered events.
## Tradeoff Analysis
The new API does not represent a breaking change of any other API, and consequently neither for the consuming applications. Only concerned-consuming applications would benefit from this new feature, while it remains entirely transparent for others.
## Decision
Provide an API to query past events of records of the given kind and return the events in paginated ascending chronological order.
{
“id”: \<RECORD_ID\>
“kind”: \<KIND\>
“op”: \<CREATE|UPDATE|DELETE, etc\>
"version": \<VERSION\>
"timestamp": \<TIMESTAMP\>\
}
## Consequences
* A new API on the Storage service would be available.
* Documentation of the Storage service should be modified with details for the new API.Yifan YeYifan Yehttps://community.opengroup.org/osdu/platform/system/storage/-/issues/184Storage Record query does not include record audit info2023-12-06T15:31:26ZAn NgoStorage Record query does not include record audit infoStorage query/records API returns records without audit information such as createdUser, createTime, modifyUser, modifyTime.
This behavior is inconsistent with other Storage record query such as the batch fetch and the record fetch APIs.Storage query/records API returns records without audit information such as createdUser, createTime, modifyUser, modifyTime.
This behavior is inconsistent with other Storage record query such as the batch fetch and the record fetch APIs.https://community.opengroup.org/osdu/platform/system/storage/-/issues/183Add a note on deleted records to /versions2023-09-06T13:20:14ZMarton NagyAdd a note on deleted records to /versions**GET /records/versions/{id}** "Get all record versions" endpoint in [Storage Service](https://p4d.developer.delfi.cloud.slb-ds.com/workspace/apiCatalog/OSDU-Storage-Service) seems to retrieve record versions regardless of the record its...**GET /records/versions/{id}** "Get all record versions" endpoint in [Storage Service](https://p4d.developer.delfi.cloud.slb-ds.com/workspace/apiCatalog/OSDU-Storage-Service) seems to retrieve record versions regardless of the record itself **being (soft) deleted** or not. While neither **GET /records/{id}** "Get record" or **GET /records/{id}/{version}** "Get record version" retrieves the record when it's (soft) deleted... doing so correctly.
Please add a note to the **GET /records/versions/{id}** endpoint description to highlight the difference.
cc @nthakur, @gehrmannhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/182Issues observed with logging2023-12-01T06:47:32ZLarissa PereiraIssues observed with logging**Issue 1: Duplicate operation IDs**
We observed multiple dependency logs for disparate operations (based on record ids) with identical operation Id's for the POST QueryApi/getRecords API. Duplicate entries were observed when reading fr...**Issue 1: Duplicate operation IDs**
We observed multiple dependency logs for disparate operations (based on record ids) with identical operation Id's for the POST QueryApi/getRecords API. Duplicate entries were observed when reading from BlobStore for operation READ_FROM_STORAGE_CONTAINER although these logs belonged to separate operations.
![image](/uploads/afc539574de597bba300b5d6b2a18b8a/image.png)
**Issue 2: Multiple dependency logs and missing Read log**
We observed multiple dependency logs with identical operation Id's for the POST QueryApi/fetchRecords. These entries were observed when querying CosmosStore, however the READ_FROM_STORAGE_CONTAINER dependency log is missing.
![image](/uploads/ce377f8bf6ee95646ca1ab5d910df167/image.png)M22 - Release 0.25VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/181GET: /records/{recordID}/{version} - ERROR 5002024-01-01T08:47:32ZSiarhei Khaletski (EPAM)GET: /records/{recordID}/{version} - ERROR 500**Context**
GET: /records/{recordID}/{version} fails with error 500 if an invalid version is provided (see the attachment)
We noticed an odd behavior of the service:
List of existing versions of the following record: `opendes:work-pro...**Context**
GET: /records/{recordID}/{version} fails with error 500 if an invalid version is provided (see the attachment)
We noticed an odd behavior of the service:
List of existing versions of the following record: `opendes:work-product-component--SamplesAnalysis:e9f02f48f43149a8b69606ff7597f391`
![image](/uploads/3d75fd80a57f5558c7d0eb00a4d795eb/image.png)
If request unexisting version `1` - status error 500
![image](/uploads/d3dc228f70263bd24ff7d09975baa63c/image.png)
Meanwhile, if request unexisting version `1234` - status 404
![image](/uploads/e82da89c3673b643aaa26845f0eb0c81/image.png)
**Azure GLab Logs**
![image](/uploads/8d54b1addcbc1835b4ea3c90135072b6/image.png)
**Expected Behavior**
404 - status codeM22 - Release 0.25Siarhei Khaletski (EPAM)Chad LeongSiarhei Khaletski (EPAM)