Storage issueshttps://community.opengroup.org/osdu/platform/system/storage/-/issues2023-04-06T20:25:51Zhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/51[Storage Service] Can't retrieve a record with encoded space symbol in ID2023-04-06T20:25:51ZDmitriy Rudko[Storage Service] Can't retrieve a record with encoded space symbol in IDThere is a **core** issue with record IDs that contain encrypted space symbols (%20):
**Create record RQ:**
```
curl --location --request PUT 'https://os-storage-attcrcktoa-uc.a.run.app/api/storage/v2/records' \
--header 'Data-Partition...There is a **core** issue with record IDs that contain encrypted space symbols (%20):
**Create record RQ:**
```
curl --location --request PUT 'https://os-storage-attcrcktoa-uc.a.run.app/api/storage/v2/records' \
--header 'Data-Partition-ID: osdu' \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data-raw '[{
"id": "osdu:work-product-component--WellboreMarkerSet:Some%20Text",
...
}]'
```
**Get record RQ:**
```
curl --location --request GET 'https://os-storage-attcrcktoa-uc.a.run.app/api/storage/v2/records/osdu:work-product-component--WellboreMarkerSet:Some%20Text' \
--header 'Data-Partition-ID: osdu' \
--header 'Authorization: Bearer <token>'
```
**Get record RS:**
```
{
"code": 400,
"reason": "Validation error.",
"message": "getLatestRecordVersion.id: Not a valid record id. Found: osdu:work-product-component--WellboreMarkerSet:Some Text"
}
```
**Expected Result:**
- I should be able to retrieve it by ID that was used during record creation
**Actual Result:**
- Its not possible to retrieve the recordM1 - Release 0.1ethiraj krishnamanaiduThomas Gehrmann [slb]Dmitriy Rudkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/42[Storage] Removal of Requirement that Data Partition ID is the first componen...2023-03-14T07:55:46ZGary Murphy[Storage] Removal of Requirement that Data Partition ID is the first component of the Kind propertyAs part of integrating with the new Schema Service (see https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/7), it was realized that a change of behavior is needed in the Storage service treatment of the "kind" ...As part of integrating with the new Schema Service (see https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/7), it was realized that a change of behavior is needed in the Storage service treatment of the "kind" property. Previously, it was required that data-partition-id was the first component of the "kind" property (i.e. data-partition-id:source:entitytype:version). However, with the Schema Service, there can be cross-partition schema authorities that should not be the data-partition-id.
An example of this would be "OSDU:wks:wellbore:<version>". Here, "OSDU" is not a data-partition-id, but an authority that spans data partition concerns. To use this Schema scoping paradigm, Storage service needs to relax the check so that <b>kind first component != data-partition-id destination</b> for the storage record.M1 - Release 0.12021-01-29https://community.opengroup.org/osdu/platform/system/storage/-/issues/164For AWS platform query to get all kinds is not returning any records.2023-03-09T21:26:03ZKamlesh TodaiFor AWS platform query to get all kinds is not returning any records.The query to retrieve all the kinds is not returning any results (records)
curl --location 'https://r3m16-ue1.preshiptesting.osdu.aws/api/storage/v2/query/kinds' \
--header 'data-partition-id: osdu' \
--header 'Accept: application/json'...The query to retrieve all the kinds is not returning any results (records)
curl --location 'https://r3m16-ue1.preshiptesting.osdu.aws/api/storage/v2/query/kinds' \
--header 'data-partition-id: osdu' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer eyJraWQiOiJ...7kPscDabFJ3sEPeNA'
The response 200 OK (with results being empty)
{
"results": []
}
The collection used can be found at https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/12_CICD_Setup_StorageAPI/Storage%20API%20CI-CD%20v1.11.postman_collection.json
The request name is "01 Storage - Get all kinds success scenario"
@chad @debasiscM16 - Release 0.19https://community.opengroup.org/osdu/platform/system/storage/-/issues/165Need example of how to use the POST /query/records:batch Fetch multiple rec...2023-03-09T21:25:28ZKamlesh TodaiNeed example of how to use the POST /query/records:batch Fetch multiple recordsThe Storage API documentation mention about
POST /query/records/batch Fetch multiple records. Would like to get the sample of how is this feature expected to be used.
Need clarification on
Account ID is the active OSDU account (OSDU ...The Storage API documentation mention about
POST /query/records/batch Fetch multiple records. Would like to get the sample of how is this feature expected to be used.
Need clarification on
Account ID is the active OSDU account (OSDU account or customer's account) which the users choose to use with the Search API.
frame-of-reference: This value indicates whether normalization applies, should be either 'none' or 'units=SI;crs=wgs84;elevation=msl;azimuth=true north;dates=utc;'
@chad @debasiscM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/storage/-/issues/77Storage service documentation - kind and id2023-03-09T18:16:32ZDebasis ChatterjeeStorage service documentation - kind and idPlease check this page -
> https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md
See how kind and id are defined. These need revision.
> id: (optional) Unique identifier in the Data ...Please check this page -
> https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md
See how kind and id are defined. These need revision.
> id: (optional) Unique identifier in the Data Ecosystem. When not provided, the service will create and assign an id to the record. Must follow the naming convention: {Data-Partition-Id}:{object-type}:{uuid}.
>
> kind: (mandatory) Kind of data being ingested. Must follow the naming convention: {Data-Partition-Id}:{dataset-name}:{record-type}:{version}.
During a recent thread with @gehrmann and @doniger , here is some clarification from Thomas.
> • id: At some point in time the term data-partition was considered alien and confusing – I started using namespace in Data Definitions. Over time the community adopted (only to some extent?) the term data-partition and the generic term namespace became alien and confusing.
> • kind: The first part is schema-authority. The (old) Storage Schema endpoints require the schema-authority to be the data-partition-id. This service is deprecated. The Schema Service has taken over and the schema-authority is now independent from the data-partition.https://community.opengroup.org/osdu/platform/system/storage/-/issues/153Indexer fetch records requests should not be checked via OPA/Policy (Or any o...2023-03-06T10:20:12ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comIndexer fetch records requests should not be checked via OPA/Policy (Or any other service, that sends internal requests)**Problem:**
Currently, the Storage service will evaluate policies for service requests of the Indexer service, which doesn't make sense since the indexer should be able to fetch any record ingested to the platform.
Indexer fetch reque...**Problem:**
Currently, the Storage service will evaluate policies for service requests of the Indexer service, which doesn't make sense since the indexer should be able to fetch any record ingested to the platform.
Indexer fetch requests use common requests authentication flow when OPA integration is enabled:
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/opa/service/OPAServiceImpl.java#L104
~~~
http://localhost:8181/v1/data/osdu/partition/osdu/dataauthz/records
{
"input": {
"operation": "view",
"token": "indexer-service-token",
"datapartitionid": "osdu",
"records": [{
"id": "osdu:master-data--Well:999907686759",
"kind": "osdu:wks:master-data--Well:1.0.0",
"legal": {
"legaltags": ["osdu-demo-legaltag"],
"otherRelevantDataCountries": ["US"],
"status": "compliant"
},
"acls": {
"viewers": ["data.default.viewers@osdu.osdu-gcp.go3-nrg.projects.epam.com"],
"owners": ["data.default.owners@osdu.osdu-gcp.go3-nrg.projects.epam.com"]
}
}
]
}
}
~~~
And it is possible that Indexer will not be authorized to fetch records:
~~~
HttpResponse(headers = {
null = [HTTP / 1.1 200 OK],
Content - Length = [305],
Date = [Tue, 29 Nov 2022 10: 58: 31 GMT],
Content - Type = [application / json]
}, body = {
"result": [{
"errors": [{
"code": 401,
"id": "osdu:master-data--Well:999907686759",
"message": "Legal response 401 {\"code\":401,\"reason\":\"Unauthorized\",\"message\":\"The user is not authorized to perform this action\"}",
"reason": "Error from compliance service"
}
],
"id": "osdu:master-data--Well:999907686759"
}
]
}, contentType = application / json, responseCode = 200, exception = null, request = http: //localhost:8181/v1/data/osdu/partition/osdu/dataauthz/records, httpMethod=POST, latency=812)
~~~
And will receive an empty response:
~~~
{
"records": [],
"notFound": [
"osdu:master-data--Well:999907686759"
],
"conversionStatuses": []
}
~~~
Which left records not indexed, and not searchable. Scenarios, when this occurrence happens, look quite easy to achieve, for example when the record uses ACLs that don't belong to the Service token.
**Solution:**
We need to bypass OPA\Policy authentication for internal service requests.M16 - Release 0.19Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRiabokon Stanislav(EPAM)[GCP]Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/8Data change notification service2023-03-02T23:20:43ZAn NgoData change notification serviceM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/storage/-/issues/158AZURE: on reading version from storage we are checking only viewer permissions2023-01-16T12:02:28ZYauheni LesnikauAZURE: on reading version from storage we are checking only viewer permissionsOn reading version from storage we are checking only viewer permissions. It would be nice to check both: viewer and owner ones.On reading version from storage we are checking only viewer permissions. It would be nice to check both: viewer and owner ones.Yauheni LesnikauYauheni Lesnikauhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/155GCP failing with core-common v0.18.0-rc42023-01-02T11:18:05ZMina OtgonboldGCP failing with core-common v0.18.0-rc4osdu-gcp-anthos-test integration tests are consistently failing when the core-common version is upgraded to v0.18.0-rc4.
Currently, gcp consumes 0.17.0 version of core-common which contains vulnerable libraries. The storage MR "Update ...osdu-gcp-anthos-test integration tests are consistently failing when the core-common version is upgraded to v0.18.0-rc4.
Currently, gcp consumes 0.17.0 version of core-common which contains vulnerable libraries. The storage MR "Update Storage to be Collaboration Context Aware" needs to consume a new version of core-common that exposes collaboration context. It is a blocker for this storage MR to be merged. As a quick fix for gcp test failure, we created a core-common that has collaboration context off of 0.17.0 version of core-common. The pipeline is passing with this version, which indicates that the gcp test failure is coming from the core-common version upgrade from 0.17.0 to 0.18.0-rc4.
References
* [Associated storage MR](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/546)
* [Core-common MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/183)
* [ADR for the storage and core-common MRs](https://community.opengroup.org/osdu/platform/system/storage/-/issues/149)Yauhen Shaliou [EPAM/GCP]Yauhen Shaliou [EPAM/GCP]https://community.opengroup.org/osdu/platform/system/storage/-/issues/151Storage service fails due to opa enabled value being true.2022-12-20T04:08:07ZNikhil Singh[MicroSoft]Storage service fails due to opa enabled value being true.2022-11-16 10:51:53.832 ERROR storage-6446654dcd-5m7cm --- [-nio-80-exec-52] o.o.o.a.l.Slf4JLogger correlation-id=fd7c531b-f76c-4467-a502-8860097b79a9 data-partition-id=opendes api-method=PUT operation-name={PUT [/reco...2022-11-16 10:51:53.832 ERROR storage-6446654dcd-5m7cm --- [-nio-80-exec-52] o.o.o.a.l.Slf4JLogger correlation-id=fd7c531b-f76c-4467-a502-8860097b79a9 data-partition-id=opendes api-method=PUT operation-name={PUT [/records], consumes [application/json], produces [application/json]} user-id=8b2a56ba-edf5-47ce-94b6-42c336ec8172 app-id=678fadf8-e5a8-46cd-a75d-4d6cc95d9bc9:storage.app error getting data authorization result {correlation-id=fd7c531b-f76c-4467-a502-8860097b79a9, data-partition-id=opendes} org.opengroup.osdu.core.common.model.http.AppException: error getting data authorization result| at org.opengroup.osdu.storage.opa.service.OPAServiceImpl.evaluateDataAuthorizationPolicy(OPAServiceImpl.java:125) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]| at org.opengroup.osdu.storage.opa.service.OPAServiceImpl.validateUserAccessToRecords(OPAServiceImpl.java:86) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]|
at org.opengroup.osdu.storage.service.IngestionServiceImpl.validateUserAccessAndCompliancePolicyConstraints(IngestionServiceImpl.java:415) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]| at org.opengroup.osdu.storage.service.IngestionServiceImpl.getRecordsForProcessing(IngestionServiceImpl.java:176) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]| at org.opengroup.osdu.storage.service.IngestionServiceImpl.createUpdateRecords(IngestionServiceImpl.java:98) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]| at org.opengroup.osdu.storage.provider.azure.service.IngestionServiceAzureImpl.createUpdateRecords(IngestionServiceAzureImpl.java:27) ~[classes!/:?]| at org.opengroup.osdu.storage.api.RecordApi.createOrUpdateRecords(RecordApi.java:80) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]| at org.opengroup.osdu.storage.api.RecordApi$$FastClassBySpringCGLIB$$495e8f0c.invoke(<generated>) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]| ... suppressed 11 lines| at org.opengroup.osdu.storage.api.RecordApi$$EnhancerBySpringCGLIB$$a32ffde7.createOrUpdateRecords(<generated>) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]| at org.opengroup.osdu.storage.api.RecordApi$$FastClassBySpringCGLIB$$495e8f0c.invoke(<generated>) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]| ... suppressed 9 lines| at org.opengroup.osdu.storage.api.RecordApi$$EnhancerBySpringCGLIB$$1ec1cefc.createOrUpdateRecords(<generated>) ~[storage-core-0.15.1-SNAPSHOT.jar!/:?]| ... suppressed 2 lines| at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]| ... suppressed 18 lines| at org.opengroup.osdu.storage.util.StorageFilter.doFilter(StorageFilter.java:86) [storage-core-0.15.1-SNAPSHOT.jar!/:?]| ... suppressed 2 lines| at org.opengroup.osdu.azure.filters.TransactionLogFilter.doFilter(TransactionLogFilter.java:74) [core-lib-azure-0.17.0-rc14.jar!/:?]| ... suppressed 34 lines| at org.opengroup.osdu.azure.filters.Slf4jMDCFilter.doFilter(Slf4jMDCFilter.java:69) [core-lib-azure-0.17.0-rc14.jar!/:?]| ... suppressed 18 lines| at com.microsoft.applicationinsights.web.internal.WebRequestTrackingFilter.doFilter(WebRequestTrackingFilter.java:142) [applicationinsights-web-2.6.4.jar!/:?]| ... suppressed 18 lines|
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]|Nikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/storage/-/issues/71Meta, Frame of Reference, CRS values2022-12-12T22:13:41ZDebasis ChatterjeeMeta, Frame of Reference, CRS valuesCan you please advise any way to avoid providing persistableReference, and simply provide short ID?
After all, the information is available from Reference data or from CRS Catalog.
The following is from CSV Ingestion step for defining C...Can you please advise any way to avoid providing persistableReference, and simply provide short ID?
After all, the information is available from Reference data or from CRS Catalog.
The following is from CSV Ingestion step for defining CRS to apply to specific columns of source CSV file.
```
"FrameOfReference": [
{
"kind": "CRS",
"name": "GCS_North_American_1927",
"persistableReference": "{\"wkt\":\"GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4267]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_North_American_1927\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4267\"},\"type\":\"LBC\"}",
"propertyNames": [
"LATITUDE",
"LONGUITUDE"
],
"propertyValues": [
"deg"
],
"uncertainty": 0
},
```https://community.opengroup.org/osdu/platform/system/storage/-/issues/134Storage and PUT - Any way to work around the limit of 500 records?2022-12-09T13:35:40ZDebasis ChatterjeeStorage and PUT - Any way to work around the limit of 500 records?I was trying to persist standard reference values for entity such as UnitOfMeasure and hit this limit.
```
{
"code": 400,
"reason": "Validation error.",
"message": "createOrUpdateRecords.records: Up to 500 records can be ing...I was trying to persist standard reference values for entity such as UnitOfMeasure and hit this limit.
```
{
"code": 400,
"reason": "Validation error.",
"message": "createOrUpdateRecords.records: Up to 500 records can be ingested at a time"
}
```
Is there something we can do for work around (apart from having to split the original JSON load manifest into smaller chunks)?
cc - @krveduru for informationhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/152Upgrade azure-storage SDK2022-11-28T14:39:21ZNur SheikhUpgrade azure-storage SDKIn storage service we are using the azure-storage sdk 8.6.5 from com.microsoft.azure package which is too old and not having much support. It iis advisable to use the latest sdk for com.azure package.In storage service we are using the azure-storage sdk 8.6.5 from com.microsoft.azure package which is too old and not having much support. It iis advisable to use the latest sdk for com.azure package.https://community.opengroup.org/osdu/platform/system/storage/-/issues/52R3 Storage service - Documentation - explanation of "kind"2022-11-21T11:47:19ZDebasis ChatterjeeR3 Storage service - Documentation - explanation of "kind"Please check the following.
`https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#create-schema`
**kind**: *(mandatory)* Kind of data being ingested. Must follow the naming conventi...Please check the following.
`https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#create-schema`
**kind**: *(mandatory)* Kind of data being ingested. Must follow the naming convention: `{Data-Partition-Id}:{dataset-name}:{record-type}:{version}`.
Suggest use "schema" rather than "dataset" for the second part. In R3, the term "Dataset" is used separately to refer to File, Collection of files etc.
See example of "kind" from GCP R3-M3 environment.
> "kind": "odesprod:wks:work-product-component--WellboreMarkerSet:1.0.0",https://community.opengroup.org/osdu/platform/system/storage/-/issues/92Provide documentation of batch records API2022-11-21T11:46:45ZDebasis ChatterjeeProvide documentation of batch records APIAs per input from @nthakur , it seems documentation is not up to date.
Neelesh sent me this link for now.
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#fetch-records
Can you...As per input from @nthakur , it seems documentation is not up to date.
Neelesh sent me this link for now.
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#fetch-records
Can you please add to main documentation?
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/api/storage_openapi.yaml
Thank youhttps://community.opengroup.org/osdu/platform/system/storage/-/issues/64[System/Storage] Storage Create/Update API seems to delete older versions aft...2022-11-21T11:40:32ZAn Ngo[System/Storage] Storage Create/Update API seems to delete older versions after update; only latest version work when fetching**Workflow:**
1. Create a new record - successfully created with record id and version.
1. Update the same record (not sure it matters, but I updated the kind of the record). New version is created.
1. Get all versions of the record; 2...**Workflow:**
1. Create a new record - successfully created with record id and version.
1. Update the same record (not sure it matters, but I updated the kind of the record). New version is created.
1. Get all versions of the record; 2 should be returned.
1. Get record version (new) - works fine.
1. Get record version (old) - 404 is returned with this error:
{
"code": 404,
"reason": "Specified blob was not found",
"message": "Status code 404, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist._RequestId:7e04d563-a01e-0026-60bc-30241a000000_Time:2021-04-13T23:25:03.2208136Z</Message></Error>\""
}
I tried updating the same record several times, and every time, it seems to delete the old version and only keeps the newest version. So If I update this record 4 times (now there are 5 versions total), then only 1 version works when fetching, the other 4 versions will fail with above error.https://community.opengroup.org/osdu/platform/system/storage/-/issues/72[Storage] Validate record parent versions in the ancestry block2022-11-21T11:35:27ZAn Ngo[Storage] Validate record parent versions in the ancestry blockWhen a derived record is created, validate that the provided parent record exists, including the version.
Use cases:
GIVEN an otherwise valid record to be PUT into storage that has an ancestry block with a parent ID: parent version
AN...When a derived record is created, validate that the provided parent record exists, including the version.
Use cases:
GIVEN an otherwise valid record to be PUT into storage that has an ancestry block with a parent ID: parent version
AND the record has a valid (existing) record ID but an invalid record version (that version does not exist)
WHEN the record is "PUT" into Storage
THEN an error is thrown indicating that the ancestry block is invalid due to an invalid record version
GIVEN a correct record to be PUT into storage that has an ancestry block with a parent ID: parent version
AND the record has a valid (existing) record ID and a valid record version
WHEN the record is "PUT" into Storage
THEN the response indicates the record is stored properly (2xx)https://community.opengroup.org/osdu/platform/system/storage/-/issues/136Schema Validation Failed - Storage Service2022-11-21T11:11:21ZSamiullah GhousudeenSchema Validation Failed - Storage ServiceData ingestion through `Storage PUT service` does not validate schema, kind & attributes.
As in below request able to ingest `"TestAttribute": "Test-Sami"` attribute/value which is not defined in Contract type reference data - WKS schem...Data ingestion through `Storage PUT service` does not validate schema, kind & attributes.
As in below request able to ingest `"TestAttribute": "Test-Sami"` attribute/value which is not defined in Contract type reference data - WKS schema.
<details><summary> Storage PUT Request </summary>
<pre><code>
curl --location --request PUT 'https://osdu-ship.msft-osdu-test.org/api/storage/v2/records' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer eyJ0eXAiOiJKV1Qi ' \
--data-raw '[
{
"id": "opendes:reference-data--ContractorType:test-sami01",
"kind": "osdu:wks:reference-data--ContractorType:1.0.0",
"acl": {
"owners": [
"data.default.owners@opendes.contoso.com"
],
"viewers": [
"data.default.viewers@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-public-usa-dataset-7643990"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"Name2": "Well",
"ID2": "Well",
"Code2": "Well",
"Source2": "Workbook Published/FacilityTypeType.1.0.0.xlsx; commit SHA 0b4db59a.",
"TestAttribute" : "Test-Sami"
}
}
]'
</code></pre>
</details>
<details><summary> Storage GET Request </summary>
<pre><code>
{
"data": {
"Name2": "Well",
"ID2": "Well",
"Code2": "Well",
"Source2": "Workbook Published/FacilityTypeType.1.0.0.xlsx; commit SHA 0b4db59a.",
"TestAttribute": "Test-Sami"
},
"meta": null,
"id": "opendes:reference-data--ContractorType:test-sami01",
"version": 1658769507968280,
"kind": "osdu:wks:reference-data--ContractorType:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-public-usa-dataset-7643990"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping@azureglobal1.onmicrosoft.com",
"createTime": "2022-07-05T17:06:37.282Z",
"modifyUser": "preshipping@azureglobal1.onmicrosoft.com",
"modifyTime": "2022-07-25T17:18:28.992Z"
}
</code></pre>
</details>
Also, able to ingest and fetch data through Storage Service without creating schema `osdu:wks:reference-data--ContractorTypeTestSami:1.0.0 ` in OSDU system as noticed below :
<details><summary> Storage PUT Request </summary>
<pre><code>
curl --location --request PUT 'https://osdu-ship.msft-osdu-test.org/api/storage/v2/records' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSU ' \
--data-raw '[
{
"id": "opendes:reference-data--ContractorTypeTestSami:test-sami01",
"kind": "osdu:wks:reference-data--ContractorTypeTestSami:1.0.0",
"acl": {
"owners": [
"data.default.owners@opendes.contoso.com"
],
"viewers": [
"data.default.viewers@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-public-usa-dataset-7643990"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"Name2": "Well",
"ID2": "Well",
"Code2": "Well",
"Source2": "Workbook Published/FacilityTypeType.1.0.0.xlsx; commit SHA 0b4db59a.",
"TestAttribute" : "Test-Sami"
}
}
]'
</code></pre>
</details>
<details><summary> Storage GET Request </summary>
<pre><code>
{
"data": {
"Name2": "Well",
"ID2": "Well",
"Code2": "Well",
"Source2": "Workbook Published/FacilityTypeType.1.0.0.xlsx; commit SHA 0b4db59a.",
"TestAttribute": "Test-Sami"
},
"meta": null,
"id": "opendes:reference-data--ContractorTypeTestSami:test-sami01",
"version": 1658770548926014,
"kind": "osdu:wks:reference-data--ContractorTypeTestSami:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-public-usa-dataset-7643990"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping@azureglobal1.onmicrosoft.com",
"createTime": "2022-07-25T17:35:49.251Z"
}
</code></pre>
</details>
cc- @chad @debasischttps://community.opengroup.org/osdu/platform/system/storage/-/issues/68[Storage / Put / Records][R3][AWS] No validation check on [schema services/ki...2022-11-21T11:11:21Zkenneth liew[Storage / Put / Records][R3][AWS] No validation check on [schema services/kind] for creation of new records.Summary:Record sucessfully created although the schema does not exist on system.
Details/Example:
`Put Method: {{osduonaws_base_url}}/api/storage/v2/records`
Body:
```
[
{
"data": {
"ID" : "ID",
...Summary:Record sucessfully created although the schema does not exist on system.
Details/Example:
`Put Method: {{osduonaws_base_url}}/api/storage/v2/records`
Body:
```
[
{
"data": {
"ID" : "ID",
"Code" : "Code",
"Description": "Description",
"Name": "Name"
},
"id": "{{data_partition_id}}:reference-data--XXX:Id_2",
"kind": "{{data_partition_id}}:wks:reference-data--XXX:1.0.0",
"legal": {
"legaltags": [
"{{data_partition_id}}-public-usa-dataset-1"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"acl": {
"viewers": [
"data.default.viewers@{{data_partition_id}}.testing.com"
],
"owners": [
"data.default.owners@{{data_partition_id}}.testing.com"
]
}
}
]
```
Status : 201 created
Response
```
{
"recordCount": 1,
"recordIds": [
"opendes:reference-data--XXX:Id_2"
],
"skippedRecordIds": [],
"recordIdVersions": [
"opendes:reference-data--XXX:Id_2:1620285305392414"
]
}
```
`Get Method :{{osduonaws_base_url}}/api/schema-service/v1/schema/{{data_partition_id}}:wks:reference-data--XXX:1.0.0`
Response:
```
{
"error": {
"code": 404,
"message": "Schema is not present",
"errors": [
{
"domain": "global",
"reason": "notFound",
"message": "Schema is not present"
}
]
}
}
```https://community.opengroup.org/osdu/platform/system/storage/-/issues/93Remove information about deprecated Schema features from Storage service tuto...2022-11-21T11:06:07ZDebasis ChatterjeeRemove information about deprecated Schema features from Storage service tutorialPlease see below. Add "Deprecated" warning here. Use of Schema service is preferred.
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#schemasPlease see below. Add "Deprecated" warning here. Use of Schema service is preferred.
https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#schemas