Storage issueshttps://community.opengroup.org/osdu/platform/system/storage/-/issues2023-06-06T20:04:08Zhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/123Storage GET record returns 404 for records with optional version (Record ID e...2023-06-06T20:04:08ZAn NgoStorage GET record returns 404 for records with optional version (Record ID ending with colon)Storage GET /api/storage/v2/records/{id} returns 404 error for records whose ID ends with a colon (version is empty).
For example, "osdu:master-data--Wellbore:nz-100000391126:"
This is the case where the version component is empty (this...Storage GET /api/storage/v2/records/{id} returns 404 error for records whose ID ends with a colon (version is empty).
For example, "osdu:master-data--Wellbore:nz-100000391126:"
This is the case where the version component is empty (this is allowed as part of [this change](https://community.opengroup.org/osdu/platform/system/storage/-/issues/26#summary-january-26-2021) in record ID validation).
Expected behavior should be returning the latest version of the record.https://community.opengroup.org/osdu/platform/system/storage/-/issues/122Storage Service Records Fetch Error2022-08-24T10:52:45ZSamiullah GhousudeenStorage Service Records Fetch Error**Not able to retrieve records from Storage Service **
If record id contains html encoded characters(%2F), then Storage service doesn't returns expected result but Search query returns as expected.
This issue same across all CSP's - A...**Not able to retrieve records from Storage Service **
If record id contains html encoded characters(%2F), then Storage service doesn't returns expected result but Search query returns as expected.
This issue same across all CSP's - AZURE, GCP, IBM & AWS.
For example , below query returns expected result through Search Service -
{
"kind": "*:wks:reference-data--UnitOfMeasure:1.0.0",
"limit": 10,
"aggregateBy":"kind",
"query":"id:\"osdu:reference-data--UnitOfMeasure:V%2FB\" OR id:\"opendes:reference-data--UnitOfMeasure:v%2Fv\" OR id:\"odesprod:reference-data--UnitOfMeasure:H%2Fm\" OR id:\"opendes:reference-data--UnitOfMeasure:US%2FF\""
}
However, in Storage service getting response - HTTP Status 400 – Bad Request
<!doctype html>
<html lang="en">
<head>
<title>HTTP Status 400 – Bad Request</title>
<style type="text/css">
body {
font-family: Tahoma, Arial, sans-serif;
}
h1,
h2,
h3,
b {
color: white;
background-color: #525D76;
}
h1 {
font-size: 22px;
}
h2 {
font-size: 16px;
}
h3 {
font-size: 14px;
}
p {
font-size: 12px;
}
a {
color: black;
}
.line {
height: 1px;
background-color: #525D76;
border: none;
}
</style>
</head>
<body>
<h1>HTTP Status 400 – Bad Request</h1>
</body>
</html>Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/system/storage/-/issues/121Storage Schema endpoints should be obsoleted2022-08-24T10:54:38ZGary MurphyStorage Schema endpoints should be obsoletedRemove code and config related to the storage schemas APIs from OSDU as they are EOL.
The following APIs are to be removed
- GET /Schema
- DELETE /Schema
- POST /schemaRemove code and config related to the storage schemas APIs from OSDU as they are EOL.
The following APIs are to be removed
- GET /Schema
- DELETE /Schema
- POST /schemahttps://community.opengroup.org/osdu/platform/system/storage/-/issues/120Inconsistent behavior of storage PUT when skipdupes is passed as true2022-08-26T10:06:09ZMandar KulkarniInconsistent behavior of storage PUT when skipdupes is passed as trueStorage PUT API has an optional query parameter called [skipdupes](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#using-skipdupes)
Current behavior of storage PUT API to update...Storage PUT API has an optional query parameter called [skipdupes](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#using-skipdupes)
Current behavior of storage PUT API to update existing record is:
If skipdupes is passed as true, if the data, meta blocks in the input request are same as the existing record content, then the record update is skipped.
When skipdupes is passed as true, the record update is skipped in a scenario when the user has passed different legal, acl, tags blocks content in the input request, but data and meta block content is same as that of the existing record.
(This happens because when skipdupes is passed as true, the storage service compares only data and meta blocks of the incoming and existing records and not all the blocks in the record.)
Expected behavior is :
If skipdupes is passed as true, both data and meta blocks should be compared. If data block is same but legal, acl, tags blocks are different, then the same record should be updated. To keep the behavior in-sync with PATCH API, the record version should not be updated in case only tags, legal or acl blocks are being changed.https://community.opengroup.org/osdu/platform/system/storage/-/issues/118[Azure] Deletion records with invalid legaltag without any logging2022-04-12T14:49:06ZYauheni Lesnikau[Azure] Deletion records with invalid legaltag without any loggingThere is an message handler which process the `LegalTagChanged ` events, and if the legal tag became invalid, appropriate records should be mark as `Inactive` (soft deleted). The issue is that there is no any logging which explicitly say...There is an message handler which process the `LegalTagChanged ` events, and if the legal tag became invalid, appropriate records should be mark as `Inactive` (soft deleted). The issue is that there is no any logging which explicitly says that deletion performs there.Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/117Storage fails to delete large number of records upon legal tag expiration2024-03-21T15:19:58ZAn NgoStorage fails to delete large number of records upon legal tag expirationIf there are large number of records associated with a legalTag that expires after running the cron job, we are seeing availability issues and inconsistent result in terms of record searchability.
**Observations:**
**LegalTag cron job...If there are large number of records associated with a legalTag that expires after running the cron job, we are seeing availability issues and inconsistent result in terms of record searchability.
**Observations:**
**LegalTag cron job update issue:**
**Scenario**: I have a large number of records (in the 6 digits) that are associated with a legalTag (i.e. the record metadata has a particular legalTag (let's call it lt1) in the legal.legaltags section). The legalTag lt1 is set to expire soon
**Event**: lt1 expires
**Action 1** : Cron job `updateLegalTagStatus` is triggered on a periodic basis, which grabs the legalTags that have changed their state (valid to invalid and invalid to valid) and publishes this information onto SB topic 'legaltags' and EG topic 'legaltagschangedtopic'. The legalTag also changes its state in the CosmosDb
'legaltagschangedtopic' has an event subscription to SB topic 'legaltagschangedtopiceg', which has a subscription 'eg_sb_legaltagssubscription'
**Action 2 **: Storage service pulls messages from 'eg_sb_legaltagssubscription' for LegalTag update events and updates records associated with lt1. Storage updates the recordMetadata with active/inactive record status and publishes the change onto SB and EG for indexer-queue to consume.
**Expected outcome:** All records associated with lt1 are now inactive. They are unsearchable from Storage and Search APIs.
**Actual outcome:** Some records associated with lt1 are now inactive. They are unsearchable from Storage and Search APIs. I can still search other records.
**Issue**: Not all records are getting pulled from Storage service at **Action 2** to be processed. Thus, many records simply don't change their state, although the legalTag is invalid now.
**Observed behavior/possible improvements:**
1. The context of legalTag change (active to inactive or inactive to active) is not considered by Storage when fetching records to update. Storage tries to fetch ALL records for that legalTag with the query
SELECT * FROM c WHERE ARRAY_CONTAINS(c.metadata.legal.legaltags, lt1). In case of large number of records, this is a longer operation. We observed throttling on the cosmos-db during this process
2. No way to retry. Because Legal service updates the letalTag status in cosmosDb, running the `updateLegalTagStatus` job again will not pick up this legal tag. To do this, we are required to manually change the status of the legalTag and run the cron job again. Upon manual retries, we face the issue above where Storage is trying to process ALL records again.
3. What happens when Storage job is interrupted, possibly due to pod restart (high cpu utilization) or network error or cosmosDb error? Retrying the whole job doesn't help muchChad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/storage/-/issues/116In Azure environment the end point to query the data with limit is not working2022-08-26T11:59:00ZKamlesh TodaiIn Azure environment the end point to query the data with limit is not workingIn the Azure environment, the end point to query the data with a limit is not working.
e.g. https://osdu-ship.msft-osdu-test.org/api/storage/v2/query/kinds?limit=10
Response: 400 Bad Request
{
"code": 400,
"reason": "Limit not suppo...In the Azure environment, the end point to query the data with a limit is not working.
e.g. https://osdu-ship.msft-osdu-test.org/api/storage/v2/query/kinds?limit=10
Response: 400 Bad Request
{
"code": 400,
"reason": "Limit not supported",
"message": "The limit is invalid"
}
@debasisc @sehuboy @kumar_vaibav @ChrisZhangM10 Patch - Release 0.13 patchKrishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/system/storage/-/issues/114No audit log for succeed PATCH updates2022-03-24T14:46:56ZYauheni LesnikauNo audit log for succeed PATCH updatesWhen the PATCH update performs there is audit logging only for failed record ids. We need to add similar logging for the succeed ones as well.When the PATCH update performs there is audit logging only for failed record ids. We need to add similar logging for the succeed ones as well.M11 - Release 0.14Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/113Storage /records endpoint without having Content-type in the header throws 41...2023-03-01T04:46:31ZAn NgoStorage /records endpoint without having Content-type in the header throws 415 errorStorage /records endpoint without having Content-type in the header throws 415 error codeStorage /records endpoint without having Content-type in the header throws 415 error codehttps://community.opengroup.org/osdu/platform/system/storage/-/issues/112Potential defect related to Delete methods due to no response body - Storage ...2022-11-21T10:12:01ZJevon WilliamsPotential defect related to Delete methods due to no response body - Storage Delete RecordDELETE API Methods that has no response body. This call will provide both a 204 success code and a 404 failure code but it does not provide any details and/or explanations:
Example 204 code is returned but it does not say item has s...DELETE API Methods that has no response body. This call will provide both a 204 success code and a 404 failure code but it does not provide any details and/or explanations:
Example 204 code is returned but it does not say item has successfully deleted.
Example 404 code is returned but it does not say why the error code was returned
URL Endpoint - base_url/api/storage/v2/records/{{recordIds}}![deleteAPI_Storage_Record_screenshot](/uploads/396a120b7291f242bf4a0fac9b1b12ed/deleteAPI_Storage_Record_screenshot.PNG)https://community.opengroup.org/osdu/platform/system/storage/-/issues/111Potential defect related to Delete methods due to no response body - Storage ...2022-11-21T10:12:09ZJevon WilliamsPotential defect related to Delete methods due to no response body - Storage SchemaDELETE API Methods that has no response body. This call will provide both a 204 success code and a 404 failure code but it does not provide any details and/or explanations:
1. Example 204 code is returned but it does not say item has su...DELETE API Methods that has no response body. This call will provide both a 204 success code and a 404 failure code but it does not provide any details and/or explanations:
1. Example 204 code is returned but it does not say item has successfully deleted.
1. Example 404 code is returned but it does not say why the error code was returned
URL Endpoint - base_url/api/storage/v2/schemas/osdu:osdu:fault-system-wp:0.2.0
![deleteAPI_Storage_screenshot](/uploads/1f100652e42805d620d8a6ee55e3dc45/deleteAPI_Storage_screenshot.PNG)https://community.opengroup.org/osdu/platform/system/storage/-/issues/110Issue in Publisher Facade2022-11-21T09:51:31ZAbhishek Kumar (SLB)Issue in Publisher FacadeThe branch is not in a running state due to bug in the azure core library.
Please refer to this issue https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/17
**Branch:** UsageOfPublishFacadeThe branch is not in a running state due to bug in the azure core library.
Please refer to this issue https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/17
**Branch:** UsageOfPublishFacadeNikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/storage/-/issues/109storage get record with version api returns 5002022-11-21T09:54:03ZNeelesh Thakurstorage get record with version api returns 500Here is a curl where the record exists and I get 200:
```
curl --location --request GET 'https://evt.api.enterprisedata.cloud.slb-ds.com/api/storage/v2/records/opendes%3Awork-product-component--RegularHeightField%3A0a16c55a-aec0-4f21-a5...Here is a curl where the record exists and I get 200:
```
curl --location --request GET 'https://evt.api.enterprisedata.cloud.slb-ds.com/api/storage/v2/records/opendes%3Awork-product-component--RegularHeightField%3A0a16c55a-aec0-4f21-a55b-abd0447d31f6/1637157881569884' \
--header 'accept: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer ***'
```
failure
Here I changed the last digit of the version:
```
curl --location --request GET 'https://evt.api.enterprisedata.cloud.slb-ds.com/api/storage/v2/records/opendes%3Awork-product-component--RegularHeightField%3A0a16c55a-aec0-4f21-a55b-abd0447d31f6/1637157881569881' \
--header 'accept: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer ***'
```
response:
```
{
"code": 500,
"reason": "Version not found",
"message": "The version 1637157881569881 can't be found for record opendes:work-product-component--RegularHeightField:0a16c55a-aec0-4f21-a55b-abd0447d31f6"
}
```
Expected result: return code is 404
Actual result: return code is 500https://community.opengroup.org/osdu/platform/system/storage/-/issues/107Intermittent record not found errors in Storage batch API2022-11-21T09:50:21ZAn NgoIntermittent record not found errors in Storage batch APIError has been reported on Storage query/records:batch API where the user sometimes is not able to retrieve a few records. The same records could be fetched at a later time. Storage service is responding with record not found error, and ...Error has been reported on Storage query/records:batch API where the user sometimes is not able to retrieve a few records. The same records could be fetched at a later time. Storage service is responding with record not found error, and is impacting 1% of the request.
**Job details to reproduce the error: **
- Number of records: 14K
- Storage batch size: 20
- Number of threads: 10
- Record size: few KBs (well head information)Neelesh ThakurNeelesh Thakurhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/106Add flexible page size option in Storage getRecordsByKind API2022-03-07T17:11:52ZVibhuti Sharma [Microsoft]Add flexible page size option in Storage getRecordsByKind APIGet records by Kind API in storage service returns results in the form of multiple pages in case of large number of records. The page size of these pages is constant - it is equal to the `limit` specified in the optional query parameter ...Get records by Kind API in storage service returns results in the form of multiple pages in case of large number of records. The page size of these pages is constant - it is equal to the `limit` specified in the optional query parameter or the default limit configuration set in config file.
# **Context**
Reindex API in indexer service calls the getRecordsByKind storage API to fetch multiple records. We need an option of querying storage without the constant page size constraint for performance related enhancements on azure provider.
# **Proposed Solution**
Add an **optional** parameter in storage service API, which when set to true will make the API return results with page size <= limit configured. The default behavior will be to return results with page size == limit configured.M10 - Release 0.13Vibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]2022-01-21https://community.opengroup.org/osdu/platform/system/storage/-/issues/104Add flexible page size option in Storage getRecordsByKind API2022-01-17T14:02:00ZVibhuti Sharma [Microsoft]Add flexible page size option in Storage getRecordsByKind APIGet records by Kind API in storage service returns results in the form of multiple pages in case of large number of records. The page size of these pages is constant - it is equal to the `limit` specified in the optional query parameter ...Get records by Kind API in storage service returns results in the form of multiple pages in case of large number of records. The page size of these pages is constant - it is equal to the `limit` specified in the optional query parameter or the default limit configuration set in config file.
# **Context**
Reindex API in indexer service calls the getRecordsByKind storage API to fetch multiple records. We need an option of querying storage without the constant page size constraint for performance related enhancements on azure provider.
# **Proposed Solution**
Add an **optional** parameter in storage service API, which when set to true will make the API return results with page size <= limit configured. The default behavior will be to return results with page size == limit configured.M10 - Release 0.13Vibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/system/storage/-/issues/103Upgrade to Log4J 2.172021-12-21T02:09:48ZDavid Diederichd.diederich@opengroup.orgUpgrade to Log4J 2.17The Apache Foundation released another Log4j2 update, version 2.17, which address a denial of service vulnerability.
This issue tracks progress to upgrade this dependency for this project.The Apache Foundation released another Log4j2 update, version 2.17, which address a denial of service vulnerability.
This issue tracks progress to upgrade this dependency for this project.https://community.opengroup.org/osdu/platform/system/storage/-/issues/102Log4J Expedient Updates and Patches2021-12-17T20:19:56ZDavid Diederichd.diederich@opengroup.orgLog4J Expedient Updates and PatchesThis issue associates MRs that were applied to this project quickly to get a patched version ready as soon as possible. The intent is to provide a reference point for later, more thoughtful, analysis.This issue associates MRs that were applied to this project quickly to get a patched version ready as soon as possible. The intent is to provide a reference point for later, more thoughtful, analysis.Spencer Suttonsuttonsp@amazon.comSpencer Suttonsuttonsp@amazon.comhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/100Storage API /query/kinds is broken and breaks reindex functionality2023-03-13T10:16:44ZGary MurphyStorage API /query/kinds is broken and breaks reindex functionality**_Takeaway_**<br/>
The /query/kinds API has been broken in OSDU Storage for quite a while, and fixing it was not a priority as Schema Service endpoints were thought to be the successor solution. This is not the case, and /query/kinds ...**_Takeaway_**<br/>
The /query/kinds API has been broken in OSDU Storage for quite a while, and fixing it was not a priority as Schema Service endpoints were thought to be the successor solution. This is not the case, and /query/kinds needs to work as designed.<br/>
**_Summary_**
The context here is the issue to change the Indexer to use Schema Service schemas instead of the original Storage Schemas (https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/7). This has been done successfully; however, the original plan to retire the Storage Schema endpoints + /query/kinds entirely exposed a hole in functionality that needs to be addressed. Essentially, it was thought that fixing /query/kinds would not be needed with the Schema Service, but the use cases where Storage is the source of truth for *in use* kinds were not caught.<br/><br/>
**Key Use Case** -- reindexing all kinds<br/><br/>
Reindexing all kinds in an Elasticsearch cluster (Reindex All) is an infrequent but vital operation. Cases where it is required include:
disaster recovery after Storage Records are restored, application of changes to Elasticsearch analyzers, and correction of indices after changes to base OSDU schemas or client schemas.<br/><br/>
Disaster Recovery Scenario:
1. All records in Storage (including underlying CosmosDB or FireStore or whatever) are brought back to RPO state.
2. The Search index is not in sync yet with the restored Storage records, so Reindex All is executed.
3. Reindex All should *not* be using the Schema get all schemas endpoint as that will retrieve every schema that has been defined in the installation which includes unused schemas and obsolete schemas and those may number in the thousands. Instead, Reindex All needs to use /query/kinds from Storage which will retrieve only those kinds actually in use in Storage.
4. As Reindex All executes, the list of kinds is retrieved from Storage /query/kinds and iterated over, triggering a reindex on each individual kind known to Storage.M10 - Release 0.13https://community.opengroup.org/osdu/platform/system/storage/-/issues/99Datetime conversion not working2022-06-09T15:51:52ZMingyang ZhuDatetime conversion not workingStorage batch API returns error for Date/Datetime conversion. Because it is not converted to UTC properly, indexer cannot process it.
Indexer error example:
{
"results": [
{
"index": {
"trace": [
"datetime ...Storage batch API returns error for Date/Datetime conversion. Because it is not converted to UTC properly, indexer cannot process it.
Indexer error example:
{
"results": [
{
"index": {
"trace": [
"datetime parsing error: unknown format for attribute: SPUD_DATE | value: 3/28/2012",
"datetime parsing error: unknown format for attribute: STATUS_DATE | value: 4/30/2018"
],
"statusCode": 400,
"lastUpdateTime": "2021-11-12T01:40:48.603Z"
},
"id": "sandbox-weu-des-prod-testing-e:wellbore:PDD-MzMwNTMwMzY2NzAxMDA"
}
],
"aggregations": null,
"totalCount": 1
}M10 - Release 0.13