OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2023-11-09T13:48:32Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/45[M18] EDS ingest fails if Search service call returns 0 records.2023-11-09T13:48:32ZDominik Kolasa[M18] EDS ingest fails if Search service call returns 0 records.FetchStartDateTime , FetchEndDateTime , Filter params of ConnectedSourceDataJob are used to filter records. If search returns 0 records:
```
INFO - Total Fetched Records: 0
ERROR - {'status': 'error', 'message': IndexError
Traceback (mos...FetchStartDateTime , FetchEndDateTime , Filter params of ConnectedSourceDataJob are used to filter records. If search returns 0 records:
```
INFO - Total Fetched Records: 0
ERROR - {'status': 'error', 'message': IndexError
Traceback (most recent call last):
File "/usr/local/airflow/dags/eds_ingest/libs/src_dags_fetch_and_ingest.py", line 98, in fetch_and_ingest
records = self._fetch_client(criteria_dict)
File "/usr/local/airflow/dags/eds_ingest/libs/src_dags_fetch_and_ingest.py", line 359, in _fetch_client
logger.info(f"Displaying only one Sample Record : {records[0]}")
```M22 - Release 0.25Jeyakumar DevarajuluJeyakumar Devarajuluhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/43[M18] EDS ingest fails on task status check2023-10-26T12:06:58ZDominik Kolasa[M18] EDS ingest fails on task status checkEDS ingest fails randomly on task status checks.
```
{{src_dags_fetch_and_ingest.py:158}} ERROR - {'status': 'error', 'message': Attribute
Traceback (most recent call last):
File "/usr/local/airflow/dags/eds_ingest/libs/src_dags_fetch...EDS ingest fails randomly on task status checks.
```
{{src_dags_fetch_and_ingest.py:158}} ERROR - {'status': 'error', 'message': Attribute
Traceback (most recent call last):
File "/usr/local/airflow/dags/eds_ingest/libs/src_dags_fetch_and_ingest.py", line 118, in fetch_and_ingest
if task_status.lower() == 'success' or current_try >= Constant.MAX_TRY:
```Priyanka BhongadePriyanka Bhongadehttps://community.opengroup.org/osdu/platform/system/register/-/issues/45M18 GET and DELETE /api/register/v1/action/{id}2023-06-15T13:35:40ZShane HutchinsM18 GET and DELETE /api/register/v1/action/{id}Received a response with 5xx status code: 500
{"timestamp":"2023-06-14T14:29:55.655+00:00","status":500,"error":"Internal Server Error","path":"/api/register/v1/action/00%3B"}
Expected a 404 error or other 4xx.
Run this get curl comman...Received a response with 5xx status code: 500
{"timestamp":"2023-06-14T14:29:55.655+00:00","status":500,"error":"Internal Server Error","path":"/api/register/v1/action/00%3B"}
Expected a 404 error or other 4xx.
Run this get curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/register/v1/action/00%3B
Run this delete curl command to reproduce this failure:
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/register/v1/action/00%25
I was able to reproduce this on AWS and Azure.
DELETE /api/register/v1/action/{id}
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/register/v1/action/00%25
GET /api/register/v1/action/{id}
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/register/v1/action/00%3Bhttps://community.opengroup.org/osdu/platform/system/register/-/issues/44M18 GET and DELETE /api/register/v1/ddms/{id}2023-06-15T14:01:56ZShane HutchinsM18 GET and DELETE /api/register/v1/ddms/{id}Received a response with 5xx status code: 500
{"timestamp":"2023-06-14T14:29:41.188+00:00","status":500,"error":"Internal Server Error","path":"/api/register/v1/ddms/00%3B"}
This should have returned a 404 or maybe a 401, but not 500.
...Received a response with 5xx status code: 500
{"timestamp":"2023-06-14T14:29:41.188+00:00","status":500,"error":"Internal Server Error","path":"/api/register/v1/ddms/00%3B"}
This should have returned a 404 or maybe a 401, but not 500.
Run this get curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/register/v1/ddms/00%3B
Run this delete curl command to reproduce this failure:
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/register/v1/ddms/00%0A
Was able to produce this issue in AWS and Azure.
GET /api/register/v1/ddms/{id}
Run this curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/register/v1/ddms/00%3B
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/register/v1/ddms/00%0Ahttps://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/issues/126M18 GET /api/entitlements/v2/groups/all2023-11-27T07:05:04ZShane HutchinsM18 GET /api/entitlements/v2/groups/allReceived a response with 5xx status code: 500
{"code":500,"reason":"Internal Server Error","message":"An unknown error has occurred"}
Thinking this might be an issue with type parameter, but this should return a 4xx like maybe 422?
Run...Received a response with 5xx status code: 500
{"code":500,"reason":"Internal Server Error","message":"An unknown error has occurred"}
Thinking this might be an issue with type parameter, but this should return a 4xx like maybe 422?
Run this curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' 'https://osdu.r3m18.preshiptesting.osdu.aws/api/entitlements/v2/groups/all?type=NONE&limit=100'
I was able to produce this issue in AWS and Azure.
GET /api/entitlements/v2/groups/all on azure:
Run this curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: opendes' 'https://osdu-ship.msft-osdu-test.org/api/entitlements/v2/groups/all?type=NONE&limit=100'https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/577M19 Azure GCZ- Unable to access registered GCZ service in AGOL2023-08-18T07:55:40ZEkta SinghM19 Azure GCZ- Unable to access registered GCZ service in AGOLSteps to replicate.
In AGOL register the item https://osdu-gcz.msft-osdu-test.org/ignite-provider/gcz/FeatureServer/1 .
A new item will be created with a unique item id. Now open this newly created item in a webmap .
Result - item will f...Steps to replicate.
In AGOL register the item https://osdu-gcz.msft-osdu-test.org/ignite-provider/gcz/FeatureServer/1 .
A new item will be created with a unique item id. Now open this newly created item in a webmap .
Result - item will fail to load
Note- Upon checking the url again of the registered item we find the service urls is getting saved ashttps://osdu-gcz.msft-osdu-test.org/ignite-provider/gcz/FeatureServer instead of https://osdu-gcz.msft-osdu-test.org/ignite-provider/gcz/FeatureServer/1M19 - Release 0.22Levi RemingtonLevi Remingtonhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/583M19: GC EDS Ingestion Error, some of the expected Reference Data is missing, ...2024-01-08T16:18:59ZAnkit GoyalM19: GC EDS Ingestion Error, some of the expected Reference Data is missing, while testing EDS and adding Connected Source Registry Entry.DAG: Osdu_ingest R3 manifest processing with providing integrity
XCom
KeyValuesaved_record_ids{}skipped_ids{'provide_manifest_integrity_task': [{'id': 'm19:master-data--ConnectedSourceRegistryEntry:AWSPreship-AGAUG102023', 'kind': 'osdu...DAG: Osdu_ingest R3 manifest processing with providing integrity
XCom
KeyValuesaved_record_ids{}skipped_ids{'provide_manifest_integrity_task': [{'id': 'm19:master-data--ConnectedSourceRegistryEntry:AWSPreship-AGAUG102023', 'kind': 'osdu:wks:master-data--ConnectedSourceRegistryEntry:1.2.0', 'reason': 'Missing parents: {SRN: m19:master-data--Organisation:AWS-PRESHIP-AGAUG102023, SRN: m19:master-data--ActivityTemplate:CSRE-ActivityTemplate-AGAUG102023, SRN: m19:reference-data--OAuth2FlowType:RefreshTokenKeyName}'}]}
@chad @debasisc @todaiksDzmitry Malkevich (EPAM)Jeyakumar DevarajuluDzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser-v2/-/issues/13M20/Azure/Preship - failure handling Trajectory data2023-10-04T02:22:40ZDebasis ChatterjeeM20/Azure/Preship - failure handling Trajectory dataUsed this source data[Trajectory-WITSML-DC.txt](/uploads/9732cc2a456502e990593b58b641229d/Trajectory-WITSML-DC.txt)
Dataset record opendes:dataset--File.Generic:b5c31b0cacdf4238b13f098beb382694
Energyml_converter runID="runId": "f2051ce...Used this source data[Trajectory-WITSML-DC.txt](/uploads/9732cc2a456502e990593b58b641229d/Trajectory-WITSML-DC.txt)
Dataset record opendes:dataset--File.Generic:b5c31b0cacdf4238b13f098beb382694
Energyml_converter runID="runId": "f2051ce6-5bfc-4374-af89-0b2582fb4b9b",
Xcom summary shows record ID =
{'record_id': 'opendes:dataset--File.Generic:5d75967ee5384e6e8f7f5f38cb78ec3b:',
Using the above dataset record, we run Manifest Ingestion by reference.
"runId": "223ec4fc-b0c5-48df-bfaa-f4fe71047ed2",
Fails
Step provide_manifest_integrity_task
[{'id': 'opendes:work-product--WorkProduct:b102ceb0-41f2-452a-8fbf-762c5497d2d7', 'kind': 'osdu:wks:work-product--WorkProduct:1.0.0', 'reason': 'Missing parents: set()'}]
See JSON file created by EnergyML_converter[Generates-manifest.json](/uploads/76464c6f8b61446cffcd42c83f466e79/Generates-manifest.json)
There are many problems with the use of trailing colon.Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser-v2/-/issues/11M20/Azure/Preship - Wellbore data type - ID created with trailing colon2023-10-03T08:44:02ZDebasis ChatterjeeM20/Azure/Preship - Wellbore data type - ID created with trailing colonIt is not common convention for ID to have trailing colon.
Please consider revision to remove this.
['opendes:work-product--WorkProduct:fc6be8ae-83f0-4394-a164-7d8786595522',
'opendes:dataset--File.WITSML:411dddd1c4bf430ca45ee31ac155e9...It is not common convention for ID to have trailing colon.
Please consider revision to remove this.
['opendes:work-product--WorkProduct:fc6be8ae-83f0-4394-a164-7d8786595522',
'opendes:dataset--File.WITSML:411dddd1c4bf430ca45ee31ac155e9e6',
'opendes:master-data--Wellbore:3c321e10-d04a-4b6d-8bcd-5eae01cab52d:']
Note Wellbore record created with trailing colon in ID.Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser-v2/-/issues/10M20/Azure/Preship - Wellbore XML file - job fails with no meaningful error me...2023-09-29T07:22:21ZDebasis ChatterjeeM20/Azure/Preship - Wellbore XML file - job fails with no meaningful error message[Wellbore21-Peter-DC.xml](/uploads/ba28d197eb9a8604d3d7de934fb70a1a/Wellbore21-Peter-DC.xml)
Provided by @pgonzalez71 .
Tried to run the workflow using this data file.
Dataset record ID = opendes:dataset--File.Generic:315d6f9b8f7f4edeb...[Wellbore21-Peter-DC.xml](/uploads/ba28d197eb9a8604d3d7de934fb70a1a/Wellbore21-Peter-DC.xml)
Provided by @pgonzalez71 .
Tried to run the workflow using this data file.
Dataset record ID = opendes:dataset--File.Generic:315d6f9b8f7f4edebb029ace3a216f99
runID = "runId": "d42c3ad3-9b4e-458a-aba5-a4daaf08a34a",
Airflow log from step of content_loading.
[Airflow-log-content-loading.txt](/uploads/21c53290e8821686acee4e15dc3a4067/Airflow-log-content-loading.txt)Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/622M20 Azure - Search periodically fails with 500 "Error making request to Polic...2023-10-19T15:15:16ZMichaelM20 Azure - Search periodically fails with 500 "Error making request to Policy service. Check the inner HttpResponse for more info." ErrorSearch periodically fails with 500 "Error making request to Policy service. Check the inner HttpResponse for more info." Error.
This issue seems to be noticed more often when multiple search requests are sent within a short period of ti...Search periodically fails with 500 "Error making request to Policy service. Check the inner HttpResponse for more info." Error.
This issue seems to be noticed more often when multiple search requests are sent within a short period of time.
I use an application that will use the search API to fetch data within a certain region. This application will send 6-15 parallel search requests at a time. The vast majority of times, this is not an issue.
In the past week we have noticed two instances where the 500 error has occurred. The first instance was on September 26 where any search request was failing, this issue seem to be present for an extended period of time (I am not sure when the issue was resolved, however, the issue was resolved the same day).
The next time we noticed an issue was when one request failed out of several and no further failed requests were noticed.
I do not think the requests that the application I am using is the cause of the issue because the vast majority of the time we don't have an issue.saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/issues/53[M20] EDS Naturalization - fails for multiple WPC records2023-11-09T13:59:05ZMarcin Brzykcy[M20] EDS Naturalization - fails for multiple WPC recordsDAG triggered manually using Airflow web UI. Input:
```plaintext
{
"execution_context":{
"id":[
"osdu:work-product-component--WellLog:MBtestEDS-63e2657a282e4d26b9e4817765287d06",
"osdu:work-product-component--...DAG triggered manually using Airflow web UI. Input:
```plaintext
{
"execution_context":{
"id":[
"osdu:work-product-component--WellLog:MBtestEDS-63e2657a282e4d26b9e4817765287d06",
"osdu:work-product-component--WellLog:MBtestEDS-93d0660cbff145c3b0df8940a2c36518",
"osdu:work-product-component--WellLog:MBtestEDS-0541b241070c417685184355bbca26f5",
"osdu:work-product-component--WellLog:MBtestEDS-19d1c04abbea4236af3509cd55af118a",
"osdu:work-product-component--WellLog:MBtestEDS-7e6a1fb41cf84ecb903192e46bd30977"
]
},
"runId":"1234"
}
```
After investigating logs in airflow web UI, error in step: update_status_finished_task was found:
```plaintext
[2023-11-03, 12:06:43 UTC] {{update_wpc_records.py:34}} INFO - In Update WPC Record --Start
[2023-11-03, 12:06:44 UTC] {{update_wpc_records.py:44}} INFO - b'{"code":400,"reason":"Bad request","message":"Cannot update the same record multiple times in the same request. Id: osdu:work-product-component--WellLog:MBtestEDS-63e2657a282e4d26b9e4817765287d06"}'
[2023-11-03, 12:06:44 UTC] {{update_wpc_records.py:45}} INFO - 400
[2023-11-03, 12:06:44 UTC] {{python.py:177}} INFO - Done. Returned value was: None
```
At the sime time, output from step generate_batches_task does not reveal any duplicates:
```plaintext
[['osdu:work-product-component--WellLog:MBtestEDS-63e2657a282e4d26b9e4817765287d06', 'osdu:work-product-component--WellLog:MBtestEDS-93d0660cbff145c3b0df8940a2c36518'], ['osdu:work-product-component--WellLog:MBtestEDS-0541b241070c417685184355bbca26f5'], ['osdu:work-product-component--WellLog:MBtestEDS-19d1c04abbea4236af3509cd55af118a'], ['osdu:work-product-component--WellLog:MBtestEDS-7e6a1fb41cf84ecb903192e46bd30977']]
```
Call to OSDU storage service shows that datasets were not naturalized
```plaintext
{
"id": "osdu:work-product-component--WellLog:MBtestEDS-7e6a1fb41cf84ecb903192e46bd30977",
"data": {
"Datasets": [
"osdu:dataset--ConnectedSource.Generic:EDS-test-7e6a1fb41cf84ecb903192e46bd30977:"
],
...
```
After our internal investigation and debugging it seems like this issue is relataed to batching records into parallel runs, where one record seems to be included in 2 batches.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/626M20 IBM - Missing TNO and VOLVE Data2024-01-29T16:54:56ZMichaelM20 IBM - Missing TNO and VOLVE DataThe TNO wpc--WellLog, wpc--WellboreTrajectory, wpc--WellboreMarker data and all VOLVE data is missing from the IBM M20 Environment.
Below is a search query that can be used to list the available data types in the IBM M20 Environment:
`...The TNO wpc--WellLog, wpc--WellboreTrajectory, wpc--WellboreMarker data and all VOLVE data is missing from the IBM M20 Environment.
Below is a search query that can be used to list the available data types in the IBM M20 Environment:
```
curl --location 'https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-search/api/search/v2/query' \
--header 'data-partition-id: opendes' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ...' \
--data '{
"kind": "osdu:wks:*:*",
"aggregateBy":"kind",
"limit":1
}'
```
Response:
```
{
"results": [
{
"data": {
"Endian": "BIG",
"Description": null,
"DatasetProperties.FileSourceInfo.ChecksumAlgorithm": null,
"DatasetProperties.FileSourceInfo.PreloadFileCreateUser": null,
"ResourceLifecycleStatus": null,
"ResourceCurationStatus": null,
"TechnicalAssuranceID": null,
"Source": null,
"EncodingFormatTypeID": "string",
"Name": "testDatasetUpload.txt",
"DatasetProperties.FileSourceInfo.Checksum": null,
"DatasetProperties.FileSourceInfo.FileSource": "5925814b43f1447ca04a2466ad4b09dd",
"ResourceHomeRegionID": null,
"DatasetProperties.FileSourceInfo.Name": "testDatasetUpload.txt",
"VirtualProperties.DefaultName": "testDatasetUpload.txt",
"Checksum": "string",
"DatasetProperties.FileSourceInfo.PreloadFilePath": null,
"ResourceSecurityClassification": null,
"ExistenceKind": null,
"DatasetProperties.FileSourceInfo.EncodingFormatTypeID": null,
"DatasetProperties.FileSourceInfo.PreloadFileModifyUser": null,
"SchemaFormatTypeID": "string"
},
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"source": "wks",
"acl": {
"viewers": [
"data.default.viewers@opendes.ibm.com"
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
},
"type": "dataset--File.Generic",
"version": 1696249568196249,
"tags": {
"dataflowId": "test-dataflowid-proxy"
},
"createTime": "2023-10-02T12:26:09.042Z",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"opendes-DatasetAPI-Legal-Tag-Test"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "osdu-bvt@osdu.opengroup.org",
"id": "opendes:dataset--File.Generic:autotest103848"
}
],
"aggregations": [
{
"key": "osdu:wks:master-data--Wellbore:1.0.0",
"count": 6486
},
{
"key": "osdu:wks:master-data--Well:1.0.0",
"count": 4950
},
{
"key": "osdu:wks:reference-data--CoordinateReferenceSystem:1.1.0",
"count": 2541
},
{
"key": "osdu:wks:reference-data--UnitOfMeasure:1.0.0",
"count": 1449
},
{
"key": "osdu:wks:master-data--Field:1.0.0",
"count": 422
},
{
"key": "osdu:wks:master-data--GeoPoliticalEntity:1.0.0",
"count": 406
},
{
"key": "osdu:wks:reference-data--CoordinateTransformation:1.1.0",
"count": 353
},
{
"key": "osdu:wks:reference-data--UnitQuantity:1.0.0",
"count": 305
},
{
"key": "osdu:wks:master-data--Organisation:1.0.0",
"count": 214
},
{
"key": "osdu:wks:reference-data--ProcessingParameterType:1.0.0",
"count": 30
},
{
"key": "osdu:wks:reference-data--StandardsOrganisation:1.0.0",
"count": 30
},
{
"key": "osdu:wks:reference-data--AliasNameType:1.0.0",
"count": 22
},
{
"key": "osdu:wks:reference-data--DrillingReasonType:1.0.0",
"count": 16
},
{
"key": "osdu:wks:reference-data--MaterialType:1.0.0",
"count": 15
},
{
"key": "osdu:wks:reference-data--TrajectoryStationPropertyType:1.0.0",
"count": 13
},
{
"key": "osdu:wks:reference-data--FacilityStateType:1.0.0",
"count": 10
},
{
"key": "osdu:wks:reference-data--VerticalMeasurementType:1.0.0",
"count": 10
},
{
"key": "osdu:wks:reference-data--CoordinateReferenceSystem:1.0.0",
"count": 8
},
{
"key": "osdu:wks:reference-data--GeoPoliticalEntityType:1.0.0",
"count": 6
},
{
"key": "osdu:wks:reference-data--VerticalMeasurementPath:1.0.0",
"count": 6
},
{
"key": "osdu:wks:reference-data--AliasNameTypeClass:1.0.0",
"count": 5
},
{
"key": "osdu:wks:reference-data--FacilityType:1.0.0",
"count": 5
},
{
"key": "osdu:wks:reference-data--FacilityEventType:1.0.0",
"count": 4
},
{
"key": "osdu:wks:reference-data--SchemaFormatType:1.0.0",
"count": 3
},
{
"key": "osdu:wks:reference-data--WellboreTrajectoryType:1.0.0",
"count": 3
},
{
"key": "osdu:wks:dataset--File.Generic:1.0.0",
"count": 2
},
{
"key": "osdu:wks:reference-data--AzimuthReferenceType:1.0.0",
"count": 2
},
{
"key": "osdu:wks:reference-data--OperatingEnvironment:1.0.0",
"count": 2
},
{
"key": "osdu:wks:master-data--Wellbore-7138148:1.0.0",
"count": 1
},
{
"key": "osdu:wks:reference-data--ResourceSecurityClassification:1.0.0",
"count": 1
},
{
"key": "osdu:wks:work-product-component--WellLog:1.1.0",
"count": 1
}
],
"totalCount": 10000
}
```https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/620M20 IBM Not able to retrieve ingested data using Search and Storage APIs2023-12-22T04:55:49ZDébora BarrettoM20 IBM Not able to retrieve ingested data using Search and Storage APIsAfter running manifest ingestion, the records can't be found using Search or Storage APIs. [M20_IBM_Manifest_ingestion_Issue_-_Debora.md](/uploads/56b160d40ca82648b5091d168eccdd18/M20_IBM_Manifest_ingestion_Issue_-_Debora.md)
- Search r...After running manifest ingestion, the records can't be found using Search or Storage APIs. [M20_IBM_Manifest_ingestion_Issue_-_Debora.md](/uploads/56b160d40ca82648b5091d168eccdd18/M20_IBM_Manifest_ingestion_Issue_-_Debora.md)
- Search response
```
{
"results": [],
"aggregations": [],
"totalCount": 0
}
```
- Storage response
```
{
"code": 500,
"reason": "error in data authorization",
"message": "error getting data authorization result"
}
```https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/619M20 - IBM - Policy Service - DataAuthz policy not working2023-09-29T12:11:46ZDadong ZhouM20 - IBM - Policy Service - DataAuthz policy not workingThe DataAuthz policy is not working. Seems the Entitlements service is not set correctly for OPA: empty permissionSet is returned.
"permissionSet": []
This will make all Storage Service and Dataset Service get/retrieve/update/delete ap...The DataAuthz policy is not working. Seems the Entitlements service is not set correctly for OPA: empty permissionSet is returned.
"permissionSet": []
This will make all Storage Service and Dataset Service get/retrieve/update/delete apis to fail. cc @chad @todaiks
![image](/uploads/0c675b1876eff1f8107bbbb36273533b/image.png)Anuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/618M20 IBM - Policy Service - Unable to load a policy2023-09-28T19:25:30ZDadong ZhouM20 IBM - Policy Service - Unable to load a policyAble to see the existing policies. But unable to load a policy with the following error returned:
```
{
"detail": "Error when talking to bundle service Unable to add policy ibm_test_partition_policy.rego to bundle server: download bund...Able to see the existing policies. But unable to load a policy with the following error returned:
```
{
"detail": "Error when talking to bundle service Unable to add policy ibm_test_partition_policy.rego to bundle server: download bundle bundle-opendes.tar.gz error: cannot unpack non-iterable NoneType object"
}
```
Please check. Thanks
![image](/uploads/ef9b72d4123f8d9745d92a2544b417c8/image.png)Anuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/625M20 IBM: Wellbore DDMS Write Welllog Bulk Data : 500: "Unexpected error and...2023-12-22T04:52:37ZChad LeongM20 IBM: Wellbore DDMS Write Welllog Bulk Data : 500: "Unexpected error and save bulk"## Description:
Getting error 500: "Unexpected error and save bulk" while trying to write bulk data
POST https://osdu-ship.msft-osdu-test.org/api/os-wellbore-ddms/ddms/v3/welllogs/opendes:work-product-component--WellLog:chad_5354/data...## Description:
Getting error 500: "Unexpected error and save bulk" while trying to write bulk data
POST https://osdu-ship.msft-osdu-test.org/api/os-wellbore-ddms/ddms/v3/welllogs/opendes:work-product-component--WellLog:chad_5354/data
## See steps to produce below at step:
- Writing welllog bulk record data
========================
Checking version wellbore DDMS
========================
Request:
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/version
Total time (seconds): 0.867531
Request body:
```json
No content
```
Response body: 200
```json
{
"service": "Wellbore DDMS OSDU",
"version": "0.2",
"buildNumber": "local",
"release": "M20",
"details": {
"environment_name": "undefined",
"cloud_provider": "ibm",
"de_client_config_timeout": "10",
"enable_read_fast_track": "False",
"read_bulk_backend": "Dask",
"write_bulk_backend": "Dask"
}
}
```
========================
Checkin health wellbore DDMS
========================
Request:
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/about
Total time (seconds): 0.881274
Request body:
```json
No content
```
Response body: 200
```json
{
"service": "Wellbore DDMS OSDU",
"version": "0.2",
"buildNumber": "local",
"cloudEnvironment": "ibm",
"release": "M20"
}
```
========================
Creating legal tag
========================
Request:
POST https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-legal/api/legal/v1/legaltags/
Total time (seconds): 1.342422
Request body:
```json
{
"name": "opendes-chad-test-1042",
"description": "Legal Tag added for Wellbore",
"properties": {
"contractId": "123456",
"countryOfOrigin": [
"US",
"CA"
],
"dataType": "Public Domain Data",
"exportClassification": "EAR99",
"originator": "Schlumberger",
"personalData": "No Personal Data",
"securityClassification": "Private",
"expirationDate": "2025-12-25"
}
}
```
Response body: 201
```json
{
"name": "opendes-chad-test-1042",
"description": "Legal Tag added for Wellbore",
"properties": {
"countryOfOrigin": [
"US",
"CA"
],
"contractId": "123456",
"expirationDate": "2025-12-25",
"originator": "Schlumberger",
"dataType": "Public Domain Data",
"securityClassification": "Private",
"personalData": "No Personal Data",
"exportClassification": "EAR99"
}
}
```
========================
Creating well record
========================
Request:
POST https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/ddms/v3/wells
Total time (seconds): 1.786024
Request body:
```json
[
{
"acl": {
"viewers": [
"data.default.viewers@opendes.ibm.com"
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
},
"data": {
"FacilityName": "Facility_chad_4838",
"ExtensionProperties": {
"UWI": "API-123"
}
},
"id": "opendes:master-data--Well:chad_4838",
"kind": "osdu:wks:master-data--Well:1.0.0",
"legal": {
"legaltags": [
"opendes-chad-test-1042"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
}
}
]
```
Response body: 200
```json
{
"recordCount": 1,
"recordIdVersions": [
"opendes:master-data--Well:chad_4838:1696324382056624"
],
"recordIds": [
"opendes:master-data--Well:chad_4838"
],
"skippedRecordIds": []
}
```
========================
Getting well record
========================
Request:
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/ddms/v3/wells/opendes:master-data--Well:chad_4838
Total time (seconds): 1.086779
Request body:
```json
No content
```
Response body: 200
```json
{
"id": "opendes:master-data--Well:chad_4838",
"version": 1696324382056624,
"kind": "osdu:wks:master-data--Well:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.ibm.com"
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
},
"legal": {
"legaltags": [
"opendes-chad-test-1042"
],
"otherRelevantDataCountries": [
"US"
]
},
"meta": null,
"data": {
"FacilityName": "Facility_chad_4838",
"ExtensionProperties": {
"UWI": "API-123"
}
},
"createTime": "2023-10-03T09:13:02.511000+00:00",
"createUser": "osdu-bvt@osdu.opengroup.org"
}
```
========================
Creating wellbore record
========================
Request:
POST https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/ddms/v3/wellbores
Total time (seconds): 1.142243
Request body:
```json
[
{
"acl": {
"viewers": [
"data.default.viewers@opendes.ibm.com"
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
},
"id": "opendes:master-data--Wellbore:chad_4838",
"kind": "osdu:wks:master-data--Wellbore:1.0.0",
"legal": {
"legaltags": [
"opendes-chad-test-1042"
],
"otherRelevantDataCountries": [
"FR",
"US"
]
},
"data": {
"ExtensionProperties": {
"airGap": {
"unitKey": "ft",
"value": 35.0
},
"country": "United States of America",
"operator": "Francois Vinyes",
"uwi": "33-089-00300-00-01",
"wellboreType": "bypass"
},
"SpatialLocation": {
"Wgs84Coordinates": {
"features": [
{
"geometry": {
"coordinates": [
[
-103.2380248,
46.8925081,
5301.0
],
[
-103.2380248,
46.8925081,
2801.0
],
[
-103.2378748,
46.892608100000004,
301.0
],
[
-103.23742477750001,
46.89270811,
-2199.0
],
[
-103.23667470999663,
46.892808120001,
-4699.0
],
[
-103.2356245974865,
46.892908130002,
-7199.0
]
],
"type": "LineString"
},
"properties": {
"name": "Newton 2-31-Lat-1"
},
"type": "Feature"
}
],
"type": "FeatureCollection"
}
},
"FacilityName": "Faciliity_chad_4838",
"WellID": "opendes:master-data--Well:chad_4838:"
},
"meta": [
{
"kind": "Unit",
"name": "Measure depth default unit",
"persistableReference": "persistableReference",
"propertyNames": [
"symbol"
]
}
]
}
]
```
Response body: 200
```json
{
"recordCount": 1,
"recordIdVersions": [
"opendes:master-data--Wellbore:chad_4838:1696324385861112"
],
"recordIds": [
"opendes:master-data--Wellbore:chad_4838"
],
"skippedRecordIds": []
}
```
========================
Getting wellbore record
========================
Request:
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/ddms/v3/wellbores/opendes:master-data--Wellbore:chad_4838
Total time (seconds): 1.06321
Request body:
```json
No content
```
Response body: 200
```json
{
"id": "opendes:master-data--Wellbore:chad_4838",
"version": 1696324385861112,
"kind": "osdu:wks:master-data--Wellbore:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.ibm.com"
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
},
"legal": {
"legaltags": [
"opendes-chad-test-1042"
],
"otherRelevantDataCountries": [
"FR",
"US"
]
},
"meta": [
{
"kind": "Unit",
"name": "Measure depth default unit",
"persistableReference": "persistableReference",
"propertyNames": [
"symbol"
]
}
],
"data": {
"ExtensionProperties": {
"airGap": {
"unitKey": "ft",
"value": 35.0
},
"country": "United States of America",
"operator": "Francois Vinyes",
"uwi": "33-089-00300-00-01",
"wellboreType": "bypass"
},
"SpatialLocation": {
"Wgs84Coordinates": {
"features": [
{
"geometry": {
"coordinates": [
[
-103.2380248,
46.8925081,
5301.0
],
[
-103.2380248,
46.8925081,
2801.0
],
[
-103.2378748,
46.892608100000004,
301.0
],
[
-103.23742477750001,
46.89270811,
-2199.0
],
[
-103.23667470999663,
46.892808120001,
-4699.0
],
[
-103.2356245974865,
46.892908130002,
-7199.0
]
],
"type": "LineString"
},
"properties": {
"name": "Newton 2-31-Lat-1"
},
"type": "Feature"
}
],
"type": "FeatureCollection"
}
},
"FacilityName": "Faciliity_chad_4838",
"WellID": "opendes:master-data--Well:chad_4838:"
},
"createTime": "2023-10-03T09:13:05.897000+00:00",
"createUser": "osdu-bvt@osdu.opengroup.org"
}
```
========================
Creating welllog record
========================
Request:
POST https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/ddms/v3/welllogs
Total time (seconds): 1.30426
Request body:
```json
[
{
"acl": {
"viewers": [
"data.default.viewers@opendes.ibm.com"
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
},
"data": {
"Curves": [
{
"CurveID": "GR_ID",
"NumberOfColumns": 1,
"Mnemonic": "GR",
"CurveUnit": "opendes:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "opendes:reference-data--LogCurveFamily:GammaRay:"
},
{
"CurveID": "POR_ID",
"NumberOfColumns": 1,
"Mnemonic": "NPOR",
"CurveUnit": "opendes:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "opendes:reference-data--LogCurveFamily:NeutronPorosity:"
},
{
"CurveID": "Bulk Density",
"NumberOfColumns": 1,
"Mnemonic": "RHOB",
"CurveUnit": "opendes:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "opendes:reference-data--LogCurveFamily:BulkDensity:"
}
],
"WellboreID": "opendes:master-data--Wellbore:chad_4838:",
"CreationDateTime": "2013-03-22T11:16:03Z",
"VerticalMeasurement": {
"VerticalMeasurement": 2680.5,
"VerticalMeasurementPathID": "opendes:reference-data--VerticalMeasurementPath:MD:",
"VerticalMeasurementUnitOfMeasureID": "opendes:reference-data--UnitOfMeasure:ft:"
},
"TopMeasuredDepth": 12345.6,
"BottomMeasuredDepth": 13856.25,
"Name": "chad_4838",
"ExtensionProperties": {
"step": {
"unitKey": "ft",
"value": 0.1
},
"dateModified": "2013-03-22T11:16:03Z"
}
},
"id": "opendes:work-product-component--WellLog:chad_4838",
"kind": "osdu:wks:work-product-component--WellLog:1.1.0",
"legal": {
"legaltags": [
"opendes-chad-test-1042"
],
"otherRelevantDataCountries": [
"US",
"FR"
]
},
"meta": [
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
"propertyNames": [
"stop.value",
"elevationReference.elevationFromMsl.value",
"start.value",
"step.value",
"reference.unitKey",
"TopMeasuredDepth",
"BottomMeasuredDepth",
"SamplingInterval"
]
},
{
"kind": "DateTime",
"name": "datetime",
"persistableReference": "{\"format\":\"yyyy-MM-ddTHH:mm:ssZ\",\"timeZone\":\"UTC\",\"type\":\"DTM\"}",
"propertyNames": [
"dateModified",
"dateCreated"
]
}
]
}
]
```
Response body: 200
```json
{
"recordCount": 1,
"recordIdVersions": [
"opendes:work-product-component--WellLog:chad_4838:1696324389357957"
],
"recordIds": [
"opendes:work-product-component--WellLog:chad_4838"
],
"skippedRecordIds": []
}
```
========================
Getting welllog record
========================
Request:
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/ddms/v3/welllogs/opendes:work-product-component--WellLog:chad_4838
Total time (seconds): 1.076804
Request body:
```json
No content
```
Response body: 200
```json
{
"id": "opendes:work-product-component--WellLog:chad_4838",
"version": 1696324389357957,
"kind": "osdu:wks:work-product-component--WellLog:1.1.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.ibm.com"
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
},
"legal": {
"legaltags": [
"opendes-chad-test-1042"
],
"otherRelevantDataCountries": [
"FR",
"US"
]
},
"meta": [
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
"propertyNames": [
"stop.value",
"elevationReference.elevationFromMsl.value",
"start.value",
"step.value",
"reference.unitKey",
"TopMeasuredDepth",
"BottomMeasuredDepth",
"SamplingInterval"
]
},
{
"kind": "DateTime",
"name": "datetime",
"persistableReference": "{\"format\":\"yyyy-MM-ddTHH:mm:ssZ\",\"timeZone\":\"UTC\",\"type\":\"DTM\"}",
"propertyNames": [
"dateModified",
"dateCreated"
]
}
],
"data": {
"Curves": [
{
"CurveID": "GR_ID",
"NumberOfColumns": 1,
"Mnemonic": "GR",
"CurveUnit": "opendes:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "opendes:reference-data--LogCurveFamily:GammaRay:"
},
{
"CurveID": "POR_ID",
"NumberOfColumns": 1,
"Mnemonic": "NPOR",
"CurveUnit": "opendes:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "opendes:reference-data--LogCurveFamily:NeutronPorosity:"
},
{
"CurveID": "Bulk Density",
"NumberOfColumns": 1,
"Mnemonic": "RHOB",
"CurveUnit": "opendes:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "opendes:reference-data--LogCurveFamily:BulkDensity:"
}
],
"WellboreID": "opendes:master-data--Wellbore:chad_4838:",
"CreationDateTime": "2013-03-22T11:16:03Z",
"VerticalMeasurement": {
"VerticalMeasurement": 2680.5,
"VerticalMeasurementPathID": "opendes:reference-data--VerticalMeasurementPath:MD:",
"VerticalMeasurementUnitOfMeasureID": "opendes:reference-data--UnitOfMeasure:ft:"
},
"TopMeasuredDepth": 12345.6,
"BottomMeasuredDepth": 13856.25,
"Name": "chad_4838",
"ExtensionProperties": {
"step": {
"unitKey": "ft",
"value": 0.1
},
"dateModified": "2013-03-22T11:16:03Z"
}
},
"createTime": "2023-10-03T09:13:09.392000+00:00",
"createUser": "osdu-bvt@osdu.opengroup.org"
}
```
========================
Writing welllog bulk record data
========================
Request:
POST https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/ddms/v3/welllogs/opendes:work-product-component--WellLog:chad_4838/data
Total time (seconds): 11.813623
Request body:
```json
{
"columns": [
"GR_ID",
"POR_ID",
"Bulk Density"
],
"index": [
0,
1,
2,
3,
4
],
"data": [
[
0,
1111.1,
2222.1
],
[
0.5,
1111.2,
2222.2
],
[
1,
1111.3,
2222.3
],
[
1.5,
1111.4,
2222.4
],
[
2,
1111.5,
2222.5
]
]
}
```
Response body: 500
```json
{
"detail": "Unexpected error and save bulk"
}
```
========================
Getting welllog record
========================
Request:
GET https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-wellbore/api/os-wellbore-ddms/ddms/v3/welllogs/opendes:work-product-component--WellLog:chad_4838
Total time (seconds): 1.192706
Request body:
```json
No content
```
Response body: 200
```json
{
"id": "opendes:work-product-component--WellLog:chad_4838",
"version": 1696324389357957,
"kind": "osdu:wks:work-product-component--WellLog:1.1.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.ibm.com"
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
},
"legal": {
"legaltags": [
"opendes-chad-test-1042"
],
"otherRelevantDataCountries": [
"FR",
"US"
]
},
"meta": [
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
"propertyNames": [
"stop.value",
"elevationReference.elevationFromMsl.value",
"start.value",
"step.value",
"reference.unitKey",
"TopMeasuredDepth",
"BottomMeasuredDepth",
"SamplingInterval"
]
},
{
"kind": "DateTime",
"name": "datetime",
"persistableReference": "{\"format\":\"yyyy-MM-ddTHH:mm:ssZ\",\"timeZone\":\"UTC\",\"type\":\"DTM\"}",
"propertyNames": [
"dateModified",
"dateCreated"
]
}
],
"data": {
"Curves": [
{
"CurveID": "GR_ID",
"NumberOfColumns": 1,
"Mnemonic": "GR",
"CurveUnit": "opendes:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "opendes:reference-data--LogCurveFamily:GammaRay:"
},
{
"CurveID": "POR_ID",
"NumberOfColumns": 1,
"Mnemonic": "NPOR",
"CurveUnit": "opendes:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "opendes:reference-data--LogCurveFamily:NeutronPorosity:"
},
{
"CurveID": "Bulk Density",
"NumberOfColumns": 1,
"Mnemonic": "RHOB",
"CurveUnit": "opendes:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "opendes:reference-data--LogCurveFamily:BulkDensity:"
}
],
"WellboreID": "opendes:master-data--Wellbore:chad_4838:",
"CreationDateTime": "2013-03-22T11:16:03Z",
"VerticalMeasurement": {
"VerticalMeasurement": 2680.5,
"VerticalMeasurementPathID": "opendes:reference-data--VerticalMeasurementPath:MD:",
"VerticalMeasurementUnitOfMeasureID": "opendes:reference-data--UnitOfMeasure:ft:"
},
"TopMeasuredDepth": 12345.6,
"BottomMeasuredDepth": 13856.25,
"Name": "chad_4838",
"ExtensionProperties": {
"step": {
"unitKey": "ft",
"value": 0.1
},
"dateModified": "2013-03-22T11:16:03Z"
}
},
"createTime": "2023-10-03T09:13:09.392000+00:00",
"createUser": "osdu-bvt@osdu.opengroup.org"
}
```Ashwani Pandeyvikas ranaAshwani Pandeyhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/issues/304M21/Azure/Preship - provided Postman collection covers limited analysis types2023-12-02T23:17:27ZDebasis ChatterjeeM21/Azure/Preship - provided Postman collection covers limited analysis typesPlease see the list.![RAFS-DDMS-V2-API](/uploads/ea49bcb7cd9e5c444486b788f653fddd/RAFS-DDMS-V2-API.PNG)
When I list available content schema, it shows more analysis/test types.
Please plan to provide revised Postman Collection.
I used t...Please see the list.![RAFS-DDMS-V2-API](/uploads/ea49bcb7cd9e5c444486b788f653fddd/RAFS-DDMS-V2-API.PNG)
When I list available content schema, it shows more analysis/test types.
Please plan to provide revised Postman Collection.
I used this collection -
https://community.opengroup.org/osdu/qa/-/blob/main/Dev/48_CICD_Setup_RAFSDDMSAPI/RAFSDDMS_API_CI-CD_v1.0.postman_collection.json?ref_type=heads
And applied some corrections as provided by @Kseniya - using that I could get clean run for some tests.
See here
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M21/Test_plan_Results_M21/RAFS%20DDMS/M21-Azure-RAFS-DDMS-V2-API-Debasis.docx
============= available content schemas
GET https://{{RAFS_DDMS_HOST}}/v2/samplesanalysis/analysistypes
Response
```
{
"rocksampleanalyses": [
"1.0.0"
],
"ccereports": [
"1.0.0"
],
"difflibreports": [
"1.0.0"
],
"transporttests": [
"1.0.0"
],
"multistageseparatortests": [
"1.0.0"
],
"compositionalanalysisreports": [
"1.0.0"
],
"swellingtests": [
"1.0.0"
],
"constantvolumedepletiontests": [
"1.0.0"
],
"wateranalysisreports": [
"1.0.0"
],
"interfacialtensiontests": [
"1.0.0"
],
"stocktankoilanalysisreports": [
"1.0.0"
],
"vaporliquidequilibriumtests": [
"1.0.0"
],
"multiplecontactmiscibilitytests": [
"1.0.0"
],
"slimtubetests": [
"1.0.0"
],
"relativepermeabilitytests": [
"1.0.0"
],
"capillarypressuretests": [
"1.0.0"
],
"fractionationtests": [
"1.0.0"
],
"extractiontests": [
"1.0.0"
],
"physicalchemistrytests": [
"1.0.0"
],
"watergasrelativepermeabilities": [
"1.0.0"
],
"rockcompressibilities": [
"1.0.0"
],
"electricalproperties": [
"1.0.0"
],
"formationresistivityindexes": [
"1.0.0"
],
"nmrtests": [
"1.0.0"
],
"multiplesalinitytests": [
"1.0.0"
],
"gcmsalkanes": [
"1.0.0"
],
"mercuryinjectionanalyses": [
"1.0.0"
],
"gcmsaromatics": [
"1.0.0"
],
"gcmsratios": [
"1.0.0"
],
"gaschromatographyanalyses": [
"1.0.0"
],
"gascompositionanalyses": [
"1.0.0"
],
"isotopes": [
"1.0.0"
],
"bulkpyrolysisanalyses": [
"1.0.0"
],
"coregamma": [
"1.0.0"
],
"uniaxial": [
"1.0.0"
]
}
```https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/678M22 Azure - Invalid legal tags found on record2024-03-14T16:16:21ZChad LeongM22 Azure - Invalid legal tags found on recordI'm trying to update the reference data to solve [issue 657](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/657). But I'm unable to create the record, invalid legal tags. Am i missing something ?
===================...I'm trying to update the reference data to solve [issue 657](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/657). But I'm unable to create the record, invalid legal tags. Am i missing something ?
========================
Creating storage record
========================
Request:
PUT https://osdu-ship.msft-osdu-test.org/api/storage/v2/records/
Total time (seconds): 1.502987
Request body:
```json
[
{
"data": {
"Name": "metre",
"NameAlias": [
{
"AliasName": "9001",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:IndustryCode:",
"DefinitionOrganisationID": "opendes:reference-data--StandardsOrganisation:Energistics.EPSG:"
},
{
"AliasName": "m",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:IndustryCode:",
"DefinitionOrganisationID": "opendes:reference-data--StandardsOrganisation:Energistics.OpenSpirit:"
},
{
"AliasName": "sm",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:IndustryCode:",
"DefinitionOrganisationID": "opendes:reference-data--StandardsOrganisation:Energistics.OpenSpirit:"
},
{
"AliasName": "m",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:IndustryCode:",
"DefinitionOrganisationID": "opendes:reference-data--StandardsOrganisation:Energistics.POSC:"
},
{
"AliasName": "m",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:IndustryCode:",
"DefinitionOrganisationID": "opendes:reference-data--StandardsOrganisation:Energistics.RP66_V1:"
},
{
"AliasName": "m",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:IndustryCode:",
"DefinitionOrganisationID": "opendes:reference-data--StandardsOrganisation:Energistics.RP66_V2:"
}
],
"ID": "m",
"InactiveIndicator": false,
"Description": "The metre is the length equal to 1 650 763.73 wavelengths in vacuum of the radiation corresponding \n\t\t\t\tto the transition between the levels 2p10 and 5d5 of the krypton-86 atom. \n\t\t\t\tIn the US, this is called a meter.\n\t\t\t",
"Code": "m",
"AttributionAuthority": "Energistics",
"AttributionPublication": "Energistics Unit of Measure Dictionary V1.0",
"AttributionRevision": "1.0",
"CoefficientA": 0,
"CoefficientB": 1,
"CoefficientC": 1,
"CoefficientD": 0,
"IsBaseUnit": true,
"PersistableReference": "{\"abcd\":{\"a\":0.0,\"b\":1.0,\"c\":1.0,\"d\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"L\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"UnitDimensionCode": "L",
"UnitDimensionName": "length",
"UnitQuantityID": "opendes:reference-data--UnitQuantity:L:",
"Source": "Workbook Published/UnitOfMeasure.1.0.0.xlsx; commit SHA 4b3c8e86.",
"CommitDate": "2021-10-30T16:40:06+02:00"
},
"meta": [],
"id": "opendes:reference-data--UnitOfMeasure:m",
"kind": "osdu:wks:reference-data--UnitOfMeasure:1.0.0",
"acl": {
"owners": [
"data.default.owners@opendes.contoso.com"
],
"viewers": [
"data.default.viewers@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-legal-chad-test"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
}
}
]
```
Response body: 400
```json
{
"code": 400,
"reason": "Invalid legal tags",
"message": "Invalid legal tags found on record"
}
```Invalid legal tags found on record"Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/690M22 AZURE - Not Able to Retrieve Subproject Metadata2024-02-15T13:56:20ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brM22 AZURE - Not Able to Retrieve Subproject MetadataHello,
I'm trying to create a **new subproject via postman** and retrieve the subproject metadata but I'm getting status 403 Forbidden (Step 5). Other thing I've noticed is when I List the subproject in a tenant, even with a legal that...Hello,
I'm trying to create a **new subproject via postman** and retrieve the subproject metadata but I'm getting status 403 Forbidden (Step 5). Other thing I've noticed is when I List the subproject in a tenant, even with a legal that in the body for the subproject creation (Step 4), the created project does not have a legal tag associated (Step6). What could be wrong?
Here is what I'm doing:
- Collection: [Azure_OSDU_Smoke_Tests.postman_collection.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Azure-M22/Services/Smoke-tests/Azure_OSDU_Smoke_Tests.postman_collection.json?ref_type=heads)
- Environment: [Verify_Preshipping_New_TeamA-E.postman_environment.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Azure-M22/Environment/Verify_Preshipping_New_TeamA-E.postman_environment.json?ref_type=heads)
## 1. Create a Legal Tag
POST https://{{LEGAL_HOST}}/legaltags
**BODY CURL VERSION**
```
curl --location 'https://osdu-ship.msft-osdu-test.org/api/legal/v1/legaltags' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer <Token>' \
--data '{
"name": "opendes-SeismicDMS-Legal-Tag-Test9295727",
"description": "Legal Tag added for VDS to VDS testing",
"properties": {
"contractId": "123456",
"countryOfOrigin": [
"US",
"CA"
],
"dataType": "Public Domain Data",
"exportClassification": "EAR99",
"originator": "Schlumberger",
"personalData": "No Personal Data",
"securityClassification": "public",
"expirationDate": "2025-12-25"
}
}'
```
**BODY**
```json
{
"name": "{{tagName}}",
"description": "Legal Tag added for VDS to VDS testing",
"properties": {
"contractId": "123456",
"countryOfOrigin": [
"US",
"CA"
],
"dataType": "Public Domain Data",
"exportClassification": "EAR99",
"originator": "Schlumberger",
"personalData": "No Personal Data",
"securityClassification": "public",
"expirationDate": "2025-12-25"
}
}
```
**RESPONSE**
```json
{
"name": "opendes-SeismicDMS-Legal-Tag-Test9295727",
"description": "Legal Tag added for VDS to VDS testing",
"properties": {
"countryOfOrigin": [
"US",
"CA"
],
"contractId": "123456",
"expirationDate": "2025-12-25",
"originator": "Schlumberger",
"dataType": "Public Domain Data",
"securityClassification": "public",
"personalData": "No Personal Data",
"exportClassification": "EAR99"
}
}
```
## 2. Register a new tenant
POST https://{{SEISMICDMS_HOST}}/tenant/{{data-partition-id}}
**BODY CURL VERSION**
```
curl --location 'https://osdu-ship.msft-osdu-test.org/seistore-svc/api/v3/tenant/opendes' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer <Token>' \
--data-raw '{
"gcpid": "AutoTest_project_id",
"esd": "opendes.contoso.com",
"default_acl": "users.datalake.admins@opendes.contoso.com"
}'
```
**BODY**
```json
{
"gcpid": "AutoTest_project_id",
"esd": "{{data-partition-id}}{{domain}}",
"default_acl": "users.datalake.admins@{{data-partition-id}}{{domain}}"
}
```
**RESPONSE**
```
[seismic-store-service] The tenant opendes has been registered already
```
## 3. Retrieve Tenant Metadata
GET https://{{SEISMICDMS_HOST}}/tenant/{{data-partition-id}}
**BODY CURL VERSION**
```
curl --location 'https://osdu-ship.msft-osdu-test.org/seistore-svc/api/v3/tenant/opendes' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer <Token>'
```
**RESPONSE**
```json
{
"name": "opendes",
"esd": "opendes.contoso.com",
"gcpid": "AutoTest_project_id",
"default_acls": "users.datalake.admins@opendes.contoso.com"
}
```
## 4. Create a new subproject
POST https://{{SEISMICDMS_HOST}}/subproject/tenant/{{data-partition-id}}/subproject/{{subprojectid}}
**BODY CURL VERSION**
```
curl --location 'https://osdu-ship.msft-osdu-test.org/seistore-svc/api/v3/subproject/tenant/opendes/subproject/subprojectodi595608' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'ltag;' \
--header 'Authorization: Bearer <Token>' \
--data-raw '{
"admin": "admin@opendes.contoso.com",
"storage_class": "MULTI_REGIONAL",
"storage_location": "US",
"acls": {
"admins": [
"data.sdms.opendes.subprojectodi595608.admin@opendes.contoso.com"
],
"viewers": [
"data.sdms.opendes.subprojectodi595608.viewer@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-SeismicDMS-Legal-Tag-Test1358259"
],
"otherRelevantDataCountries": [
"US"
]
}
}'
```
**BODY**
```
{
"admin": "admin@{{data-partition-id}}{{domain}}",
"storage_class": "MULTI_REGIONAL",
"storage_location": "US",
"acls": {
"admins": [
"data.sdms.{{autotestTenantid}}.{{subprojectid}}.admin@{{data-partition-id}}{{domain}}"
],
"viewers": [
"data.sdms.{{autotestTenantid}}.{{subprojectid}}.viewer@{{data-partition-id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{tagName}}"
],
"otherRelevantDataCountries": [
"US"
]
}
}
```
**RESPONSE**
```
PASS Status code is 200
PASS Status description - OK
subprojectid - subprojectodi595608 (Additional Information, was not in the response body)
```
## 5. Retrieve subproject metadata
GET https://{{SEISMICDMS_HOST}}/subproject/tenant/{{data-partition-id}}/subproject/{{subprojectid}}
**BODY CURL VERSION**
```
curl --location 'https://subproject/tenant/opendes/subproject/subprojectodi595608' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer <Token>'
```
**RESPONSE**
```
[seismic-store-service] User not authorized to perform this operation
Status: 403 Forbiden
```
## 6. List subproject in a tenant
GET https://{{SEISMICDMS_HOST}}/subproject/tenant/{{data-partition-id}}
**BODY CURL VERSION**
```
curl --location 'https://osdu-ship.msft-osdu-test.org/seistore-svc/api/v3/subproject/tenant/opendes' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer <Token>'
```
**RESPONSE**
```json
{
"name": "subprojectodi595608",
"tenant": "opendes",
"ltag": "",
"acls": {
"admins": [
"data.sdms.opendes.subprojectodi595608.admin@opendes.contoso.com"
],
"viewers": [
"data.sdms.opendes.subprojectodi595608.viewer@opendes.contoso.com"
]
},
"access_policy": "uniform",
"enforce_key": true,
"gcs_bucket": "ss-cloud-kwgbwij7gyjh488"
}
```
Regards,
JulianaM22 - Release 0.25Om Prakash GuptaOm Prakash Gupta