Pre Shipping issueshttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues2024-03-20T15:49:56Zhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/362Azure - Manifest Ingestion - Performance Load testing 50k records - 413 response2024-03-20T15:49:56ZSamiullah GhousudeenAzure - Manifest Ingestion - Performance Load testing 50k records - 413 responseGetting below error while submitting request for performance load testing 50k records -
`DEBUG:root:Response: 413
DEBUG:root:json = {"timestamp": 1662643629785, "status": 413, "error": "Payload Too Large", "message": "Request content e...Getting below error while submitting request for performance load testing 50k records -
`DEBUG:root:Response: 413
DEBUG:root:json = {"timestamp": 1662643629785, "status": 413, "error": "Payload Too Large", "message": "Request content exceeded limit of 2000 kB", "path": "/api/workflow/v1/workflow/Osdu_ingest/workflowRun"}`
fyi, successfully able to complete performance load testing for 500 & 1000 records.M15 - Release 0.18Srinivasan NarayananOm Prakash GuptaSrinivasan Narayananhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/635M21 Policy validation issue for non-ascii characters2024-03-14T16:27:36ZDadong ZhouM21 Policy validation issue for non-ascii charactershttps://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/122
In M21, the new policy validation has issue for non-ascii characters. Here is a sample policy tested in M21 GC:
```
package osdu.partition["m19"]....https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/issues/122
In M21, the new policy validation has issue for non-ascii characters. Here is a sample policy tested in M21 GC:
```
package osdu.partition["m19"].organisation_code_2
organisation_code := {
"AGÊNCIA NACIONAL DO PETRÓLEO": {
"Name": "ANP",
"Code": "G0013"
}
}
```
Failed to load the policy with the following error:
```
{
"detail":"Unable to validate policy! Error: {\n \"code\": \"invalid_parameter\",\n \"message\": \"error(s) occurred while compiling module(s)\",\n \"errors\": [\n {\n \"code\": \"rego_parse_error\",\n \"message\": \"unexpected assign token: expected rule value term (e.g., organisation_code := \<VALUE\> { ... })\",\n \"location\": {\n \"file\": \"tmp/m19/organisation_code_2.rego\",\n \"row\": 3,\n \"col\": 19\n },\n \"details\": {\n \"line\": \"organisation_code := {\",\n \"idx\": 18\n }\n }\n ]\n}\n 400."
}
```
This policy can be loaded before the policy validation is added in M21.
Please fix it for M22. Thanks.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/664RI Manifest Ingestion Payload More Than 10 MB Fails2024-02-26T15:53:07ZNaufal Mohamed NooriRI Manifest Ingestion Payload More Than 10 MB FailsI did perform manifest ingestion with bulk and batch manifest ingestion with huge size payload, the Airflow process failed with the following log:
`*** No logs found on s3 for ti=<TaskInstance: Osdu_ingest.process_manifest_task_2 07a625...I did perform manifest ingestion with bulk and batch manifest ingestion with huge size payload, the Airflow process failed with the following log:
`*** No logs found on s3 for ti=<TaskInstance: Osdu_ingest.process_manifest_task_2 07a62538-1c1e-4200-9d63-f880bde99362 [failed]>
*** Could not read served logs: [Errno -2] Name or service not known`
The list of runID wiht huge payload (more than 10 MB):
- 07a62538-1c1e-4200-9d63-f880bde99362 (batch 8000 records IDs payload)
- a1a8100b-d061-45e4-941d-9b85711d44ec (bulk 16000 records IDs payload)
I have performed a smaller payload (less than 10mb json) and they were successful
Do slack me in person or via Preship channel I can share the json payload in person (I cant attach the json payload here due to limit of 10mb attachment)
cc @davidglass @debasiscDenis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/568M19 RI Wellbore DDMS (NoSuchKey) when calling the GetObject operation2024-02-26T14:38:27ZBhuwan Prasad UpadhyayM19 RI Wellbore DDMS (NoSuchKey) when calling the GetObject operationPostman Environment: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M19/QA_Artifacts_M19/envFilesAndCollections/envFiles/OSDU%20R3%20M19%20RI%20Pre-ship.postman_environment.json
Postman Collection: https://comm...Postman Environment: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M19/QA_Artifacts_M19/envFilesAndCollections/envFiles/OSDU%20R3%20M19%20RI%20Pre-ship.postman_environment.json
Postman Collection: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M19/QA_Artifacts_M19/envFilesAndCollections/Wellbore%20DDMS%20CI-CD%20v3.0.postman_collection.json
1. get OSDU WellLog data
```bash
curl --location 'https://osdu.bm-preship.gcp.gnrg-osdu.projects.epam.com/api/os-wellbore-ddms/ddms/v3/welllogs/osdu:work-product-component--WellLog:AutoTest_999125827554/data' \
--header 'data-partition-id: osdu' \
--header 'offset: 0' \
--header 'limit: 100' \
--header 'curves;' \
--header 'describe: false' \
--header 'orient: split' \
--header 'accept: application/json' \
--header 'Authorization: Bearer <token>'
```
2. commit WellLog session
```bash
curl --location --request PATCH 'https://osdu.bm-preship.gcp.gnrg-osdu.projects.epam.com/api/os-wellbore-ddms/ddms/v3/welllogs/osdu:work-product-component--WellLog:AutoTest_999125827554/sessions/null' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <token>' \
--data '{
"state": "commit"
}'
```
3. Get OSDU Trajectory data
```
curl --location 'https://osdu.bm-preship.gcp.gnrg-osdu.projects.epam.com/api/os-wellbore-ddms/ddms/v3/wellboretrajectories/osdu:work-product-component--WellboreTrajectory:AutoTest_999125827554/data?curves=Measured%20Depth%2CAzimuth' \
--header 'data-partition-id: osdu' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <token>'
```
Above curl commands output response (status code 500):
```
{
"error": [
"An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist."
]
}
```Yan Sushchynski (EPAM)YannickDzmitry Malkevich (EPAM)Yan Sushchynski (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/643In R3M21 Pre-ship RI environment the endpoint to patch commit WellLog session...2024-02-26T14:37:24ZKamlesh TodaiIn R3M21 Pre-ship RI environment the endpoint to patch commit WellLog session for Wellbore DDMS is not failing with 500 Internal Server ErrorIn R3M21 Pre-ship RI environment the endpoint to patch commit WellLog session for Wellbore DDMS is not failing with 500 Internal Server Error.
<details><summary>Request to commit wellog session</summary>
curl --location --request PATCH ...In R3M21 Pre-ship RI environment the endpoint to patch commit WellLog session for Wellbore DDMS is not failing with 500 Internal Server Error.
<details><summary>Request to commit wellog session</summary>
curl --location --request PATCH 'https://osdu.bm21.gcp.gnrg-osdu.projects.epam.com/api/os-wellbore-ddms/ddms/v3/welllogs/osdu:work-product-component--WellLog:AutoTest_999714120507/sessions/null' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJhbGciOiJSUzI1Ni...Truncated...JzRkR1sswQtS-Mtg' \
--data '{
"state": "commit"
}'
</details>
<details><summary>Response for the above request</summary>
Response 500 Internal Server Error
{
"error": [
"Access Denied."
]
}
</details>
The same request from the same collection is working in GC and Azure environments
The collection can be found [here](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M21/QA_Artifacts_M21/envFilesAndCollections/Wellbore%20DDMS%20CI-CD%20v3.0.postman_collection.json?ref_type=heads)M22 - Release 0.25Kamlesh TodaiYan Sushchynski (EPAM)Dzmitry Malkevich (EPAM)Kamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/609GC CLOUD - M20 - WITSML WELL Parser Ingestion Issue2024-02-26T14:29:24ZAnkit GoyalGC CLOUD - M20 - WITSML WELL Parser Ingestion IssueThe WITSML WELL record is not ingesting, the job is failing.
{
"workflowId": "ab22864f-eb4b-4117-aa73-9501dea8efc7",
"runId": "622876de-5c42-4da2-998b-2dc40b7e293b",
"startTimeStamp": 1695661581275,
"endTimeStamp": 16956...The WITSML WELL record is not ingesting, the job is failing.
{
"workflowId": "ab22864f-eb4b-4117-aa73-9501dea8efc7",
"runId": "622876de-5c42-4da2-998b-2dc40b7e293b",
"startTimeStamp": 1695661581275,
"endTimeStamp": 1695661766970,
"status": "failed",
"submittedBy": "preshipping_test_user_m19@gcp.gnrg-osdu.projects.epam.com"
}
[2023-09-25, 17:09:27 UTC] {taskinstance.py:1776} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/osdu_airflow/operators/update_status.py", line 158, in execute
raise PipelineFailedError("Dag failed")
osdu_ingestion.libs.exceptions.PipelineFailedError: Dag failed
[2023-09-25, 17:09:27 UTC] {taskinstance.py:1279} INFO - Marking task as FAILED. dag_id=Energistics_xml_ingest, task_id=update_status_finished_task, execution_date=20230925T170621, start_date=20230925T170924, end_date=20230925T170927
[2023-09-25, 17:09:27 UTC] {standard_task_runner.py:93} ERROR - Failed to execute job 293565 for task update_status_finished_task (Dag failed; 21319)
[2023-09-25, 17:09:28 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2023-09-25, 17:09:28 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule checkKamlesh TodaiDzmitry Malkevich (EPAM)Ankit GoyalKamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/686AZURE M22 - Core Services - Unable to convert Unit without use of persistable...2024-02-15T13:55:29ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brAZURE M22 - Core Services - Unable to convert Unit without use of persistable referenceHello,
I`m trying to complete Core Services - FoR (Unit Conversion) without use of persistableReference but when I search the record the unit is not converted. So far I've:
- Collection: [Azure_OSDU_Smoke_Tests.postman_collection.json]...Hello,
I`m trying to complete Core Services - FoR (Unit Conversion) without use of persistableReference but when I search the record the unit is not converted. So far I've:
- Collection: [Azure_OSDU_Smoke_Tests.postman_collection.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Azure-M22/Services/Smoke-tests/Azure_OSDU_Smoke_Tests.postman_collection.json?ref_type=heads)
- Environment: [Verify_Preshipping_New_TeamA-E.postman_environment.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Azure-M22/Environment/Verify_Preshipping_New_TeamA-E.postman_environment.json?ref_type=heads)
## 1. Create a Record
PUT https://{{STORAGE_HOST}}/records/
**BODY CURL VERSION**
```
curl --location --request PUT 'https://osdu-ship.msft-osdu-test.org/api/storage/v2/records/' \
--header 'data-partition-id: opendes' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <Token>' \
--header 'Content-Type: application/json' \
--data-raw '[
{
"acl": {
"owners": [
"data.default.owners@opendes.contoso.com"
],
"viewers": [
"data.default.viewers@opendes.contoso.com"
]
},
"data": {
"DefaultVerticalMeasurementID": "Rotary Table",
"FacilityEvents": [
{
"EffectiveDateTime": "2002-05-12T00:00:00",
"FacilityEventTypeID": "opendes:reference-data--FacilityEventType:SPUD:"
},
{
"EffectiveDateTime": "2002-06-07T00:00:00",
"FacilityEventTypeID": "opendes:reference-data--FacilityEventType:DRILLING%20FINISH:"
}
],
"FacilityID": "A14-02-test",
"FacilityOperators": [
{
"FacilityOperatorOrganisationID": "opendes:master-data--Organisation:NAM:"
}
],
"FacilityStates": [
{
"FacilityStateTypeID": "opendes:reference-data--FacilityStateType:Abandoned:"
}
],
"FacilityTypeID": "opendes:reference-data--FacilityType:A14-02:",
"GeoContexts": [
{
"GeoPoliticalEntityID": "opendes:master-data--GeoPoliticalEntity:Netherlands:",
"GeoTypeID": "opendes:reference-data--GeoPoliticalEntityType:Country:"
},
{
"GeoPoliticalEntityID": "opendes:master-data--GeoPoliticalEntity:A14:",
"GeoTypeID": "opendes:reference-data--GeoPoliticalEntityType:BlockID:"
}
],
"NameAliases": [
{
"AliasName": "A14-02-test",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:WELL_NAME:"
},
{
"AliasName": "8577-test",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:UWI:"
}
],
"OperatingEnvironmentID": "opendes:reference-data--OperatingEnvironment:OFF:",
"Source": "TNO",
"SpatialLocation": {
"Wgs84Coordinates": {
"features": [
{
"geometry": {
"coordinates": [
3.39488769,
55.32443529
],
"type": "Point"
},
"properties": {},
"type": "Feature"
}
],
"type": "FeatureCollection"
}
},
"VerticalMeasurements": [
{
"VerticalCRSID": "opendes:reference-data--CoordinateReferenceSystem:MSL:",
"VerticalMeasurement": 39.2,
"VerticalMeasurementID": "Rotary Table",
"VerticalMeasurementPathID": "opendes:reference-data--VerticalMeasurementPath:Elevation:",
"VerticalMeasurementUnitOfMeasureID": "opendes:reference-data--UnitOfMeasure:M:"
}
]
},
"id": "opendes:master-data--Well:JFA29JANAZURE",
"kind": "osdu:wks:master-data--Well:1.0.0",
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-4375119"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [
{
"kind": "Unit",
"name": "ft",
"unitOfMeasureID": "opendes:reference-data--UnitOfMeasure:ft:",
"propertyNames": [
"VerticalMeasurements[].VerticalMeasurement"
]
}
],
"version": 0
}
]'
```
**BODY**
```json
[
{
"acl": {
"owners": [
"{{New_OwnerDataGroup}}@{{data-partition-id}}{{domain}}"
],
"viewers": [
"{{New_ViewerDataGroup}}@{{data-partition-id}}{{domain}}"
]
},
"data": {
"DefaultVerticalMeasurementID": "Rotary Table",
"FacilityEvents": [
{
"EffectiveDateTime": "2002-05-12T00:00:00",
"FacilityEventTypeID": "{{data-partition-id}}:reference-data--FacilityEventType:SPUD:"
},
{
"EffectiveDateTime": "2002-06-07T00:00:00",
"FacilityEventTypeID": "{{data-partition-id}}:reference-data--FacilityEventType:DRILLING%20FINISH:"
}
],
"FacilityID": "A14-02-test",
"FacilityOperators": [
{
"FacilityOperatorOrganisationID": "{{data-partition-id}}:master-data--Organisation:NAM:"
}
],
"FacilityStates": [
{
"FacilityStateTypeID": "{{data-partition-id}}:reference-data--FacilityStateType:Abandoned:"
}
],
"FacilityTypeID": "{{data-partition-id}}:reference-data--FacilityType:A14-02:",
"GeoContexts": [
{
"GeoPoliticalEntityID": "{{data-partition-id}}:master-data--GeoPoliticalEntity:Netherlands:",
"GeoTypeID": "{{data-partition-id}}:reference-data--GeoPoliticalEntityType:Country:"
},
{
"GeoPoliticalEntityID": "{{data-partition-id}}:master-data--GeoPoliticalEntity:A14:",
"GeoTypeID": "{{data-partition-id}}:reference-data--GeoPoliticalEntityType:BlockID:"
}
],
"NameAliases": [
{
"AliasName": "A14-02-test",
"AliasNameTypeID": "{{data-partition-id}}:reference-data--AliasNameType:WELL_NAME:"
},
{
"AliasName": "8577-test",
"AliasNameTypeID": "{{data-partition-id}}:reference-data--AliasNameType:UWI:"
}
],
"OperatingEnvironmentID": "{{data-partition-id}}:reference-data--OperatingEnvironment:OFF:",
"Source": "TNO",
"SpatialLocation": {
"Wgs84Coordinates": {
"features": [
{
"geometry": {
"coordinates": [
3.39488769,
55.32443529
],
"type": "Point"
},
"properties": {},
"type": "Feature"
}
],
"type": "FeatureCollection"
}
},
"VerticalMeasurements": [
{
"VerticalCRSID": "{{data-partition-id}}:reference-data--CoordinateReferenceSystem:MSL:",
"VerticalMeasurement": 39.2,
"VerticalMeasurementID": "Rotary Table",
"VerticalMeasurementPathID": "{{data-partition-id}}:reference-data--VerticalMeasurementPath:Elevation:",
"VerticalMeasurementUnitOfMeasureID": "{{data-partition-id}}:reference-data--UnitOfMeasure:M:"
}
]
},
"id": "{{data-partition-id}}:master-data--Well:JFA29JANAZURE",
"kind": "osdu:wks:master-data--Well:1.0.0",
"legal": {
"legaltags": [
"{{LegalTagNameExists}}"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [
{
"kind": "Unit",
"name": "ft",
"unitOfMeasureID": "{{data-partition-id}}:reference-data--UnitOfMeasure:ft:",
"propertyNames": [
"VerticalMeasurements[].VerticalMeasurement"
]
}
],
"version": 0
}
]
```
**RESULTS**
```json
{
"recordCount": 1,
"recordIds": [
"opendes:master-data--Well:JFA29JANAZURE"
],
"skippedRecordIds": [],
"recordIdVersions": [
"opendes:master-data--Well:JFA29JANAZURE:1706539464473285"
]
}
```
## 2. Search - Retrieve the record
POST https://{{SEARCH_HOST}}/query
**BODY CURL VERSION**
```
curl --location 'https://osdu-ship.msft-osdu-test.org/api/search/v2/query' \
--header 'Authorization: Bearer <Token>' \
--header 'data-partition-id: opendes' \
--header 'Content-Type: application/json' \
--data '{
"kind": "osdu:wks:master-data--Well:*.*.*",
"query": "id: \"opendes:master-data--Well:JFA29JANAZURE\" "
}
'
```
**BODY**
```json
{
"kind": "osdu:wks:master-data--Well:*.*.*",
"query": "id: \"opendes:master-data--Well:JFA29JANAZURE\" "
}
```
**RESULTS**
```json
{
"results": [
{
"data": {
"FacilityStates": [
{
"FacilityStateTypeID": "opendes:reference-data--FacilityStateType:Abandoned:"
}
],
"VirtualProperties.DefaultLocation.QuantitativeAccuracyBandID": null,
"ResourceLifecycleStatus": null,
"DefaultVerticalMeasurementID": "Rotary Table",
"SpatialLocation.Wgs84Coordinates": {
"geometries": [
{
"coordinates": [
3.39488769,
55.32443529
],
"type": "point"
}
],
"type": "geometrycollection"
},
"ResourceCurationStatus": null,
"TechnicalAssuranceID": null,
"VirtualProperties.DefaultLocation.SpatialGeometryTypeID": null,
"FacilityName": null,
"Source": "TNO",
"FacilityID": "A14-02-test",
"VirtualProperties.DefaultName": null,
"VerticalMeasurements": [
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": "opendes:reference-data--CoordinateReferenceSystem:MSL:",
"VerticalReferenceID": null,
"VerticalMeasurementSourceID": null,
"VerticalMeasurementID": "Rotary Table",
"VerticalMeasurementPathID": "opendes:reference-data--VerticalMeasurementPath:Elevation:",
"VerticalMeasurement": 39.2,
"VerticalMeasurementTypeID": null,
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": "opendes:reference-data--UnitOfMeasure:M:",
"VerticalReferenceEntityID": null
}
],
"VirtualProperties.DefaultLocation.CoordinateQualityCheckPerformedBy": null,
"VersionCreationReason": null,
"ResourceSecurityClassification": null,
"InterestTypeID": null,
"DataSourceOrganisationID": null,
"SpatialLocation.SpatialParameterTypeID": null,
"ExistenceKind": null,
"SpatialLocation.CoordinateQualityCheckPerformedBy": null,
"FacilityOperators": [
{
"FacilityOperatorID": null,
"FacilityOperatorOrganisationID": "opendes:master-data--Organisation:NAM:"
}
],
"UWI-JFA": null,
"FacilityTypeID": "opendes:reference-data--FacilityType:A14-02:",
"NameAliases": [
{
"AliasName": "A14-02-test",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:WELL_NAME:",
"DefinitionOrganisationID": null
},
{
"AliasName": "8577-test",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:UWI:",
"DefinitionOrganisationID": null
}
],
"DefaultVerticalCRSID": null,
"FacilityEvents": [
{
"EffectiveDateTime": "2002-05-12T00:00:00+0000",
"FacilityEventTypeID": "opendes:reference-data--FacilityEventType:SPUD:"
},
{
"EffectiveDateTime": "2002-06-07T00:00:00+0000",
"FacilityEventTypeID": "opendes:reference-data--FacilityEventType:DRILLING%20FINISH:"
}
],
"VirtualProperties.DefaultLocation.IsDecimated": false,
"TechnicalAssuranceTypeID": null,
"GeoContexts": [
{
"BasinID": null,
"FieldID": null,
"PlayID": null,
"GeoPoliticalEntityID": "opendes:master-data--GeoPoliticalEntity:Netherlands:",
"GeoTypeID": "opendes:reference-data--GeoPoliticalEntityType:Country:",
"ProspectID": null
},
{
"BasinID": null,
"FieldID": null,
"PlayID": null,
"GeoPoliticalEntityID": "opendes:master-data--GeoPoliticalEntity:A14:",
"GeoTypeID": "opendes:reference-data--GeoPoliticalEntityType:BlockID:",
"ProspectID": null
}
],
"CurrentOperatorID": null,
"SpatialLocation.QualitativeSpatialAccuracyTypeID": null,
"SpatialLocation.SpatialGeometryTypeID": null,
"OperatingEnvironmentID": "opendes:reference-data--OperatingEnvironment:OFF:",
"VirtualProperties.DefaultLocation.SpatialParameterTypeID": null,
"ResourceHomeRegionID": null,
"VirtualProperties.DefaultLocation.QualitativeSpatialAccuracyTypeID": null,
"InitialOperatorID": null,
"VirtualProperties.DefaultLocation.Wgs84Coordinates": {
"geometries": [
{
"coordinates": [
3.39488769,
55.32443529
],
"type": "point"
}
],
"type": "geometrycollection"
},
"SpatialLocation.QuantitativeAccuracyBandID": null
},
"kind": "osdu:wks:master-data--Well:1.0.0",
"source": "wks",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"type": "master-data--Well",
"version": 1706539464473285,
"tags": {
"normalizedKind": "osdu:wks:master-data--Well:1"
},
"createTime": "2024-01-29T14:44:25.103Z",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-4375119"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping@azureglobal1.onmicrosoft.com",
"id": "opendes:master-data--Well:JFA29JANAZURE"
}
],
"aggregations": null,
"phraseSuggestions": [],
"totalCount": 1
}
```
The **"VerticalMeasurement": 39.2** is not converted. Can you help me with that?
Regards,
Juliana FernandesM22 - Release 0.25VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/687Issue in workflow to convert from SEGY to OpenVDS2024-02-08T13:54:02ZMohd Asad ShaikhIssue in workflow to convert from SEGY to OpenVDSI am trying to convert segy to openvds via sdutil.
When I check the Dag it gives an error. : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=segy-to-vds-conversion&task_id=upd[…]xecution_date=2024-01-29T17%3A12%3A38.170763%2B00...I am trying to convert segy to openvds via sdutil.
When I check the Dag it gives an error. : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=segy-to-vds-conversion&task_id=upd[…]xecution_date=2024-01-29T17%3A12%3A38.170763%2B00%3A00M22 - Release 0.25Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/573RI: File service end-to-end workflow scenario2024-02-08T12:25:22ZChad LeongRI: File service end-to-end workflow scenario# Summary
This is a post to summarize the existing behavior of the File service end-to-end workflow scenario across the platform providers today and aims to provide a standardized behavior.
A user should be able to securely upload and ...# Summary
This is a post to summarize the existing behavior of the File service end-to-end workflow scenario across the platform providers today and aims to provide a standardized behavior.
A user should be able to securely upload and retrieve the file from the OSDU data platform.
## Steps to reproduce
1. Generate signed URL
| RI | AWS | Azure | GC | IBM |
| ------ | ------ | ------ | ------ | ------ |
| `GET https://osdu.bm-preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/uploadURL` | `GET https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/uploadURL` | `GET https://osdu-ship.msft-osdu-test.org/api/file/v2/files/uploadURL` | `GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/uploadURL` | `GET https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-file/api/file/v2/files/uploadURL` |
| ```{"FileID": "a547911d165342efb2b1a0a2b0d62be0","Location": {"SignedURL": "https://s3.bm-preship.gcp.gnrg-osdu.projects.epam.com/refi-osdu-staging-area/b694b155-ab58-4219-90e1-62ad05340c1c/a547911d165342efb2b1a0a2b0d62be0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fileUser%2F20230807%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230807T203924Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=a985ad6e9c96797e3200b780e557c04f5147dc6b35de1f00fc6701926543df79","FileSource": "/b694b155-ab58-4219-90e1-62ad05340c1c/a547911d165342efb2b1a0a2b0d62be0"}}``` | ```{ "FileID": "624d080b600641abaf3af970f6312676", "Location": { "SignedURL": "https://psosdu-shared-file-20230607030747431500000005.s3.us-east-2.amazonaws.com/osdu/mngFQIw0uAnFGTfASsb6A33xFDOfzmo6/624d080b600641abaf3af970f6312676?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMiJHMEUCIQD6O%2F3HiYMc22n0NjW33UfkP6LynnyOKftkcSFSJNGFUAIgNtWeF0ziMXd1vlqqENBI6%2BcXayHxYD%2Fl%2FG6GVg%2B%2BFKEqpgUIeBACGgw1NjE3MzUyOTE0MjciDNFmocR6p9x5wb8WIiqDBYyB7XWL7yAZNo27Fa7mCF9L4TJfDky%2Fd5QvUYE9QKp7x60DP7GtC%2Bh1%2Fo0wOykyp1PKKcsjmiFN97%2BZkLhtwvCAHRnIrlt%2FObnOU6RnPORGm73yZTMxe94iGjCfmgrM0ac6HZx2GI7SCnGs9%2FOpfSNi75z4jXPDH6RIjLMx3iHexLU3sM6n0Dn5aA8sAy0xYcUmOfsyGBnFgQpN3Szw6P8NjnrsqnwIlua%2BK3AhMgVAHucJud7ORPXhdnkOph8U%2FH1ePfCphtg7H6ub2seVJA%2B%2B9brAvg87uR4Txxg%2F%2F9LGit%2FM6AEJqlVmgMeUDj234WJkB4zp6i96etxH7UOjSYbAZ7%2Bs35f4oSZNwtsbtyiamCo75XzMam%2BLGCiFRQ0KDJ7oS9lk7A3gh6dWx6dxObxb1UrXSjrMTCrQkMMn5p9tm5MYxgF9288Eb%2BlRNM%2FiUOP1SVhvLgzDlbWGWtdHIH7c2PuzRNJxc4pK0pWxGA2z3QbBdFKTmlLGuEnzZbONXKXJ4JdUTp%2FKqpEipclYqqcAFBbF2xr6sBQ6PIAZJ%2FQ5JT1ZkFDA8nuHVVOEJvuuifAXskHn3FDGG3NtEGO%2Fn4uKi%2FisCXIi1u7yJ%2B9WDoDzJ0VB5s0vTkXslppR23T%2BXtLI%2FXOkW6QOgiCeZviS40bPM6sSJFdV6w0TdpkMcUVxmLuOytpXnNNhig%2FlQJQA6aaXIbHJe8NgAce8nVFxlxlI4xI9PUwtgXSw7Xat0cHu19tiAoFAVap%2Bvf6f7HN5oaFuzVjxvgB1D%2BGZczt84RhR2OIQR%2FR13zOT0H05qTctk40PoXYv%2FrVTDIu1xlfJ0joboH01zMfUXHglb3g0t5ahIX8w6JTEpgY6nQGV8ZN9rthZJHdpIpwAKTNM%2Fxd7vut0OsO0%2BFNm6KmP749WiLSSt%2BHIyzV2ggbd7YTc2kwrxLeYzn8q9538aVhdWb8uxPC6DdaEbcZBZcE0%2BZ%2BdlXROyLz7YiGFkGrzxFBZ4ss1xJIKanirDui9P9tZ%2Fiz3OIyN%2FsrVFQMPcqNhHslor7I2ZgKEGGDz285gLCqxDExSfaORuxLmtS%2FJ&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230807T151448Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86399&X-Amz-Credential=ASIAYFSQIKIRV7YB25NL%2F20230807%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=2c4bcbc19b45e9fdf7faba7314885189ea013e81d61ad91805dfa83bfa49248c", "FileSource": "s3://psosdu-shared-file-20230607030747431500000005/osdu/mngFQIw0uAnFGTfASsb6A33xFDOfzmo6/624d080b600641abaf3af970f6312676"}}``` | ```{ "FileID": "91018f5735bf46ff907a3f9657584daa", "Location": { "SignedURL": "https://osdumvpdp1shipbbngdata.blob.core.windows.net/file-staging-area/osdu-user%2F1691481200990-2023-08-08-07-53-20-990%2F91018f5735bf46ff907a3f9657584daa?sv=2020-08-04&st=2023-08-08T07%3A53%3A21Z&se=2023-08-08T08%3A53%3A21Z&skoid=04e80323-039e-48fe-b76b-3a65c3cf7bcf&sktid=58975fd3-4977-44d0-bea8-37af0baac100&skt=2023-08-08T07%3A53%3A21Z&ske=2023-08-08T08%3A53%3A21Z&sks=b&skv=2020-08-04&sr=b&sp=cw&sig=j0cJNQ21TOcCBltbmuymaTaJiMssi7bXMqcgjwNzWuI%3D", "FileSource": "/osdu-user/1691481200990-2023-08-08-07-53-20-990/91018f5735bf46ff907a3f9657584daa" }}``` | ```{ "FileID": "1ba9be427638426aba6e3ebaa6109690", "Location": { "SignedURL": "https://storage.googleapis.com/osdu-data-prod-odesprod-staging-area/b33afab6-2865-4347-b76e-1071ca738bb2/1ba9be427638426aba6e3ebaa6109690?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=workload-file-gcp%40osdu-service-prod.iam.gserviceaccount.com%2F20230807%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20230807T111802Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=c5d6854dc74ccc8a0d4fdcfb97a09b7e22a93e866d1478262534a8fe31546ed4cd26cca7a8202c0c2f3b2a5d6d4d1191180b02089e1a3c641fc56cd58e80e33694158919e68953a797f5743da6f7628efa4142662d8dd93adbb2c1bff6eddb1332a96ddf63e19e3296532bea31f334212bec8ab49b5eb15c1cc97ce790288e6c393771314ee831586cd32af8e7a7071f0d2c11b26e6193a67b57553268a2ad753090275e7e4ec201223c7821b3a7c7c241564324817abed2c4bb0b8307fb471a0d34d400bf285356194e2c88fc4abffe9c277f3cdd34f160a02849439017dc83b34de214585a1dcc78b1b15129041301d4047ba69c1df7cfc1fa094af79658cb", "FileSource": "/b33afab6-2865-4347-b76e-1071ca738bb2/1ba9be427638426aba6e3ebaa6109690" }}``` | ```{ "FileID": "899f355b50204081acac0b4fcde57fa5", "Location": { "SignedURL": "https://minio-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/oc-cpd-opendes-staging-bucket/899f355b50204081acac0b4fcde57fa5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230808T075530Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=minio%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=35cd4e12e801fd7873145ff245db43f91ec648801c0e17c6abd927a0c15bd354", "FileSource": "899f355b50204081acac0b4fcde57fa5" }}``` |
2. Uploading to signed URL
| RI | AWS | Azure | GC | IBM |
| ------ | ------ | ------ | ------ | ------ |
| PUT https://s3.bm-preship.gcp.gnrg-osdu.projects.epam.com/refi-osdu-staging-area/922679a4-48d3-4c47-9d23-ad12e7628825/e6bb3aff28d2429eb6f2928fc40e1a67?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fileUser%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T075128Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4e2107d3337b161a2c788dae8180c8346fffb60371954f19faefebbda93a8a | PUT https://psosdu-shared-file-20230607030747431500000005.s3.us-east-2.amazonaws.com/osdu/BbdSbYXtJnxF8F5BfW8fngl0ObIHsTzO/81723dde071042e5ba9799511c1b972e?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOD%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMiJIMEYCIQD%2FjYrMAjvArc07IY%2FbbVYvMu%2BTle2D%2BerSd%2BJQcGvovwIhANhjmuDHAqo8BzbYyjzPMHGzHIGVZwqhmYJvJGae%2FNbnKrEFCIn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAhoMNTYxNzM1MjkxNDI3Igyh%2BEWujuBzMHUHRzMqhQVlnW6tD3Uk%2BUTESlXVmcEJ6F4lcy4Wdg1r2ubSy%2FspwH14tH32Keey4%2FFOXCmTLntSa3eXs%2FmWIa%2BZzrlNDC6MW6eJyl0zKq%2F375%2BdbRbLGPUqhoB1T8rJ4F2g7bCnhPceWeLjdAbzrKbZBFIrHo%2BhsAhH8XdmzdnstyerDC6dhu7kpK88eV07WdjF%2F5WcMVccZ1GjkzE7X%2BjXrPiXByaRdymXJ2wwnwvgNqLC0tD%2FeeMkOlIBB4sHDYDiACTz%2FtMiQ7ZH0LO7a0wgy%2Fq0c%2BI7R%2Fg2PeFPUDHYF%2BzuFI%2BVvvaeWBqKS%2BA9rKVStGlWw0rWFpevBggBnbW6IDhMyLR7GlEfIyHOFGG9XOP0uGzbkdcVJcWq2az3AjuS85h%2F7n3EQD5xV5Q0cwYlt9YJUS47B%2FQoAea%2B%2FvIGzM7hngDSuBC5jk0BGrL1CjA8%2B30bXWdkdBRGgAOrAGHXf9e3knZcKW%2B39Te2g0sN0cTODVqQ8iOACc6HdI%2B6rhe1pH2I7AbehawgkSwzC8qDB2PfTvVkIvu8OIwthQny2ErX%2FZNJqGMsi4T9fDCz61kAMnviTLgAUCNygyGdpi3J0B9MU6QI%2FEdKh9fCVACZ1SiO56DgLnKEHC5pCbER%2Fkl%2Bb2KBwCyOFaMvqHiJ9Alvm9343wyFGGtwlSDcXntWS8nRj2CxmyCxj2njZaVm9iVTkShJHnJJp5MklQIj7Q7bjVJAM05Bt%2BXq2QdDzlJUfP9z9eksMVQCNy8wQI%2FZe4BF2I27NnT7dg7tSeF47YoncGfnd1%2BqrDYW8JGoCY%2BqKlIbC2PC6TN9rXU1La316F5PjFdYveq1e22EYMnRCvm9CJZEyU1HGd%2Bv674w5%2BjHpgY6nAHJlzjr%2FLR0jOObAdYCLhEzYFHgxTKPdd6CWhs%2BJsOediQRRa4Cv2IxoIsN7FvKFvKUsicc2%2BcNcfUBvGTR2N%2BzKso8OZ8ClfuRCBe4QP35MULOpYBIWuLBsUh%2FQTyIQ8jgxxa86cf49SO%2B8QrJjD%2Bq6CvznaZCCf9HX8qLkYw%2BtdHG5xERl6fSJJNPp%2FQmlufgb86j9s7LLkcwp0Y%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230808T075311Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86399&X-Amz-Credential=ASIAYFSQIKIR7FZCMBHI%2F20230808%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=275927b6df4f7487e2921d05ca054526ffbdeb02ce989d35eca2a59f8da9a967 | PUT https://osdumvpdp1shipbbngdata.blob.core.windows.net/file-staging-area/osdu-user%2F1691481200990-2023-08-08-07-53-20-990%2F91018f5735bf46ff907a3f9657584daa?sv=2020-08-04&st=2023-08-08T07%3A53%3A21Z&se=2023-08-08T08%3A53%3A21Z&skoid=04e80323-039e-48fe-b76b-3a65c3cf7bcf&sktid=58975fd3-4977-44d0-bea8-37af0baac100&skt=2023-08-08T07%3A53%3A21Z&ske=2023-08-08T08%3A53%3A21Z&sks=b&skv=2020-08-04&sr=b&sp=cw&sig=j0cJNQ21TOcCBltbmuymaTaJiMssi7bXMqcgjwNzWuI%3D | PUT https://storage.googleapis.com/osdu-data-prod-odesprod-staging-area/68bb9081-443a-45ac-bb9d-eb21aac553fd/0bd8018e8d674cb9a6c958aeb6b4f114?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=workload-file-gcp%40osdu-service-prod.iam.gserviceaccount.com%2F20230808%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20230808T075836Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=3aceeef67678f39a57db8744defaedb2f48ec4634df991ba513adf7204190c3fb9b032997b5149939cb7dd89d6acc58f81ae2e6d4142fa4f44cf8d4b36515d332ff01c94bcb2529231334587fa9401752d708ee4e308f09e6404c61a4ec5590ff70a44757c7cc081a049734e3d71f79706024837bc47f06b621a69e845a8acf7850c7b5babc5504c145de560df68bd6ee37cb559ee95562447bb95857b17766e3f9e35efbbc7d14e3e309e4f246cb928ff04a16a6e660934ecdb58abc665f40789738682ed85c845afafcd49a70c73767235927d35e2a04e56447f653f91aed7e3aef400a9986e0fe8afddcd19a89410584f183d28ed03247cddb21ead1608a9 | PUT https://minio-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/oc-cpd-opendes-staging-bucket/899f355b50204081acac0b4fcde57fa5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230808T075530Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=minio%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=35cd4e12e801fd7873145ff245db43f91ec648801c0e17c6abd927a0c15bd354 |
3. Create file metadata
| RI | AWS | Azure | GC | IBM |
| ------ | ------ | ------ | ------ | ------ |
| ```POST https://osdu.bm-preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/metadata``` | ```POST https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/metadata``` | ```POST https://osdu-ship.msft-osdu-test.org/api/file/v2/files/metadata``` | ```POST https://preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/metadata``` | ```POST https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-file/api/file/v2/files/metadata``` |
Request Body Example:
<details><summary>Click to expand</summary>
```json
{
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"meta": [
{
"name": {
"id": [
{
"kind": "12587932"
}
]
}
}
],
"tags": {
"dataflowId": "dataflowId",
"dataflowId-2": "dataflowId"
},
"acl": {
"owners": [
"data.default.owners@osdu.group"
],
"viewers": [
"data.default.viewers@osdu.group"
]
},
"legal": {
"legaltags": [
"osdu-FileAPI-Legal-Tag-Test"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"ResourceHomeRegionID": "osdu:reference-data--OSDURegion:AWSEastUSA:",
"ResourceHostRegionIDs": [
"namespace:reference-data--OSDURegion:AWSEastUSA:"
],
"ResourceCurationStatus": "osdu:reference-data--ResourceCurationStatus:CREATED:",
"ResourceLifecycleStatus": "osdu:reference-data--ResourceLifecycleStatus:LOADING:",
"ResourceSecurityClassification": "osdu:reference-data--ResourceSecurityClassification:RESTRICTED:",
"Source": "Example Data Source",
"ExistenceKind": "osdu:reference-data--ExistenceKind:Prototype:",
"Name": "Well_Headers_and_Completion_Reports 2017.csv",
"Description": "As originally delivered by ACME.com.",
"TotalSize": "1324521",
"EncodingFormatTypeID": "osdu:reference-data--EncodingFormatType:text%2Fcsv:",
"SchemaFormatTypeID": "osdu:reference-data--SchemaFormatType:CWLS%20LAS3:",
"Endian": "BIG",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "/922679a4-48d3-4c47-9d23-ad12e7628825/e6bb3aff28d2429eb6f2928fc40e1a67",
"PreloadFilePath": "s3://staging-area/r7/raw-data/provided/documents/1000.witsml",
"PreloadFileCreateUser": "somebody@acme.org",
"PreloadFileCreateDate": "2019-12-16T11:46:20.163Z",
"PreloadFileModifyUser": "somebody.else@acme.org",
"PreloadFileModifyDate": "2019-12-20T17:20:05.356Z",
"Name": "tesing,copy.txt",
"FileSize": "95463",
"EncodingFormatTypeID": "osdu:reference-data--EncodingFormatType:application%2Fgeo%2Bjson:",
"Checksum": "checksum",
"ChecksumAlgorithm": "SHA-256"
}
},
"ExtensionProperties": {
"Classification": "Raw File",
"Description": "An text further describing this file example.",
"ExternalIds": [],
"FileDateCreated": {},
"FileDateModified": {},
"FileContentsDetails": {
"TargetKind": "slb:NewZealandCSV:wellbore:1.0.0",
"FileType": "csv",
"FrameOfReference": [
{
"kind": "CRS",
"name": "GCS_WGS_1984",
"persistableReference": "{\"wkt\":\"GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4326]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_WGS_1984\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4326\"},\"type\":\"LBC\"}",
"propertyNames": [
"LATITUDE",
"LONGITUDE"
],
"propertyValues": [
"deg"
],
"uncertainty": 0
},
{
"kind": "DateTime",
"persistableReference": "{\"type\": \"DAT\", \"format\": \"MM-dd-yyyy\"}",
"propertyNames": [
"PERMIT_DATE",
"INITIAL_COMPLETION_DATE",
"STATUS_DATE",
"SPUD_date"
],
"propertyValues": [],
"uncertainty": 0
},
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
"propertyNames": [
"MD",
"TVD",
"ELEVATION"
],
"propertyValues": [
"ft"
],
"uncertainty": 0
}
]
}
}
}
}
```
</details>
Response body: 201
```json
{
"id": "osdu:dataset--File.Generic:c79b2ec7-83c0-486b-b413-b3bf5536f316"
}
```
4. Generating a signed url to download file
| RI | AWS | Azure | GC | IBM |
| ------ | ------ | ------ | ------ | ------ |
| GET https://osdu.bm-preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/osdu:dataset--File.Generic:d505785d-479e-4bfa-a5f4-28c37a8e8ec2/downloadURL?expiryTime=15M | GET https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/osdu:dataset--File.Generic:c79b2ec7-83c0-486b-b413-b3bf5536f316/downloadURL?expiryTime=15M | GET https://osdu-ship.msft-osdu-test.org/api/file/v2/files/opendes:dataset--File.Generic:459be362-facb-4c44-b805-7da4e165b0a6/downloadURL?expiryTime=15M | GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/odesprod:dataset--File.Generic:39d4cc35-5135-4b10-b742-95d838b43638/downloadURL?expiryTime=15M | GET https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-file/api/file/v2/files/opendes:dataset--File.Generic:00602949-656b-4f99-a523-c29c100dac6c/downloadURL?expiryTime=15M |
| ```{ "SignedUrl": "https://s3.bm-preship.gcp.gnrg-osdu.projects.epam.com/refi-osdu-persistent-area/922679a4-48d3-4c47-9d23-ad12e7628825/e6bb3aff28d2429eb6f2928fc40e1a67?response-content-type=text%2Fplain&response-content-disposition=attachment%3B%20filename%3Dtesing%2Ccopy.txt&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fileUser%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T075134Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=ddfbd5830f80ad8e9f2aa53c095bf0dee73c1de63653e72280cc87943c18cfef"}``` | ```{ "SignedUrl": "https://psosdu-shared-file-20230607030747431500000005.s3.us-east-2.amazonaws.com/osdu/BbdSbYXtJnxF8F5BfW8fngl0ObIHsTzO/81723dde071042e5ba9799511c1b972e?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOD%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMiJHMEUCICHSjHHYquPNhPauPr%2Bj0mLbTnzrbbO0LROgfIUCybUyAiEA0Y7urQSEoGkj%2F6tMRijNIe%2FrIvy5kwzdUvuCwLdw8rEqkAUIif%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw1NjE3MzUyOTE0MjciDFn%2FI6Ebin8xMtVwKSrkBLNAzhK0JceuhqmBoQ%2Ba3zCpuyXDR6NTkIv8RfEu0Qe%2BtommPQgBbU%2Fp6NqrHFVD28EUvlUNmTRS7O49qHnKFeRv5o536oWQK0dTUw2LVcl4%2BLMgO5rHYasaywLeJw4AesrRYf3sEvDAxAdjrvlC9yzoi27bTvvJeswaHo2D7TLFq2lM%2BY8CWWfW5zl%2FY7%2FNWnnZsZ%2B2G37LEvq0c9vxdvxJBX58uADlqS8Msrl%2F1HN514NQN8rGTAzvCPEXujnnHsQse7Pj%2FkC6lGBs5xzYF6quRiAkB7kyMMM%2F2nMkkX5v%2BUef7JWS2NKVdr4jwGaSvBsVhyquINXJFtdUMHsAbiB8SfHpap5pWKTw%2BNE75ZRHzFXpyVcXbWaT1V95UhViHCaOmwGfvrB4dJizriUTlBaqZbe1Uva9uvfuCc1zf1B%2Fi4ojCpEF2kzcvRd1J3HbDoeTBn1GHy92OudDnP7xbdlCkOBb3oHBqfN%2Bnut2V0VLF1BFnPDehog3w6Uu5zhaabAsvy6fBtNhALElZXBHdSBEByCh4aL47kNT67dIsmJ4B7BuxQ5htuHIzyLT99tkCQqoEJajhkZCtkcugkV8sGID7WpLcn8KngEZb0x0TJ6vknUScS72I3UZHt53JNXkqKY4tlAZwpe24LNkJtlnuqJVLimjRawRm6%2FhYSUiewwHEJuOp%2FcuOqtBo8GrqcxoSx15iuzGAPW35ERv22n64SptkPAUOnsrn5yH%2Banui3U5SfT6SWVfQlwYZZWy5UthZdAMctxEx0FxdpdJzU8JryqKidl8caVLDhWgFcW%2FhUIbgyk3UDDu6MemBjqdAdraWXyjQJ2n5wNnzGlOxR0rMcwThmV5xdKakoXcyrSbQArcioyRTFf%2FHmrw%2B3y1XYdcAe5roe1kIPg%2FLEogp7MSErYCt52LbBoQDByh8xUuj5lIqYUEsiKL%2BtDfJH%2BMmo5z9hcSs7uneEfWAkR3SCpJqsQMB87Kqr2cFisZeNY6%2F2Ce77Mn%2F7zxnfnudUCF4TrynwVeW4Ysydym1Fg%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230808T075318Z&X-Amz-SignedHeaders=host&X-Amz-Expires=899&X-Amz-Credential=ASIAYFSQIKIR6PYRBPWH%2F20230808%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=8e516594ad4be350c7b1bc9ad7f86e58407c3003c7aeec813ac25ea0218e72de"}``` | ```{ "SignedUrl": "https://osdumvpdp1shipbbngdata.blob.core.windows.net/file-persistent-area/osdu-user%2F1691481200990-2023-08-08-07-53-20-990%2F91018f5735bf46ff907a3f9657584daa?sv=2020-08-04&se=2023-08-08T08%3A08%3A29Z&skoid=04e80323-039e-48fe-b76b-3a65c3cf7bcf&sktid=58975fd3-4977-44d0-bea8-37af0baac100&skt=2023-08-08T07%3A53%3A29Z&ske=2023-08-08T08%3A08%3A29Z&sks=b&skv=2020-08-04&sr=b&sp=r&sig=lKcTIxIW1iyFrJxK0bwPdzbt7%2FlbSsOBD8Q%2FIP79HyE%3D&rscd=attachment%3B%20filename%3D%20tesing%2Ccopy.txt&rsct=text%2Fplain"}``` | ```{ "SignedUrl": "https://storage.googleapis.com/osdu-data-prod-odesprod-persistent-area/68bb9081-443a-45ac-bb9d-eb21aac553fd/0bd8018e8d674cb9a6c958aeb6b4f114?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=workload-file-gcp%40osdu-service-prod.iam.gserviceaccount.com%2F20230808%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20230808T075840Z&X-Goog-Expires=900&X-Goog-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dtesing%2Ccopy.txt&response-content-type=text%2Fplain&X-Goog-Signature=b0803ed77d01ba22a356a8c44e211f5d99024b07c6d70fd3738315f3e94470f841506b36802bba44cd95ec96ea0f9f9b4141fc834f30eee12040ec859dbbfe16c30264845120deb0564162b7252b750fa98dd7f46f6309efff5b9b99f57f099cbda1f9c3c4d5cd05fe422e76b1878e927e3a0ea3165dc17e4e169f52676ada2bd8ab826c915087d08c1184f3f8be61bdad8d1e1647334cd4d0f30b1f4ff2b14eaca9f5c5ef8051b8089d92f6ce28996f84eca4d30713272fdbdae5fa611cb0b3b4514246a1ce5c303d788dc0f732320132b65f133721e37196236a17ce41a6e4c22bc4d53f273064c752c499bded8be44bfaa2edca20c1ab120f9fa90bb2613c"}``` | ```{ "SignedUrl": "https://minio-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/oc-cpd-opendes-persistent-bucket/899f355b50204081acac0b4fcde57fa5?response-content-disposition=attachment%3B%20filename%20%3D%22tesing%2Ccopy.txt%22&response-content-type=text%2Fplain&response-expires=15M&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230808T075542Z&X-Amz-SignedHeaders=host&X-Amz-Expires=899&X-Amz-Credential=minio%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=009d8ac0e613b42d740cc2e7562b38e4d7ff54bf5445561d57025d1c3efe59a0"}``` |
## Intended Behavior
The downloaded file should honor and preserve the `Name` attribute in the metadata as per the [schema](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Authoring/abstract/AbstractFileSourceInfo.1.0.0.json#L49) and in our test scenario, return the following file as `tesing,copy.txt` (A non-standard file name with a comma has been chosen for validation purposes)
```json
{
...
"DatasetProperties": {
"FileSourceInfo": {
"Name": "tesing,copy.txt"
}
}
...
}Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/667Erroneous Versioning Endpoints in GC Reference Implementation (RI) M222024-02-08T12:25:02ZTamas KerekjartoErroneous Versioning Endpoints in GC Reference Implementation (RI) M22After running the Versioning Endpoints v0.1postman collection I found issues with the following services. Please see the attached report [Versioning_Endpoints_v0.1-2024-01-17-19-36-49-341-0.html](/uploads/60f7cab43894a5970a4bfb0c5ad48436...After running the Versioning Endpoints v0.1postman collection I found issues with the following services. Please see the attached report [Versioning_Endpoints_v0.1-2024-01-17-19-36-49-341-0.html](/uploads/60f7cab43894a5970a4bfb0c5ad48436/Versioning_Endpoints_v0.1-2024-01-17-19-36-49-341-0.html):
- DataSet Version
- Notification Version
- Workflow Version
- File Version
- CRS Catalog Version
I used the [Versioning Endpoints v0.1.postman_collection](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/GC-M22/Versioning%20Endpoints%20v0.1.postman_collection.json?ref_type=heads) from the GC-M22 folder.Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/688AZURE M22 - Ingestion Services: manifest processing with providing integrity2024-02-07T11:05:11ZValentin GauthierAZURE M22 - Ingestion Services: manifest processing with providing integrityI tried to ingest a manifest (attached file [test-resqml-geosiris-azure.json](/uploads/345027d7aef7fd763ec4ee27251a4666/test-resqml-geosiris-azure.json), I faced an issue during the "manifest processing with providing integrity" operator...I tried to ingest a manifest (attached file [test-resqml-geosiris-azure.json](/uploads/345027d7aef7fd763ec4ee27251a4666/test-resqml-geosiris-azure.json), I faced an issue during the "manifest processing with providing integrity" operator task.
The reason of a skipped ids is : `'reason': "'recordIds'"`. The others 'Missing parents' reasons are related to the first not ingested element (opendes:work-product-component--LocalBoundaryFeature:5fa99eb4-b11f-4f08-b1be-2d64ff14286f), because other WPC refer to it.
*By the way I think it is strange the 'missing parent' makes the ingestion fail because a WPC with this id is allready present in the catalog*
Link of the airflow dag run xcom values : [link](https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest_by_reference&task_id=process_single_manifest_file_task&execution_date=2024-01-19T18%3A27%3A41.994248%2B00%3A00)
Hypotheses:
- The input manifest has other errors that makes the ingestion fail (and maybe the 'reason' is not too specific to understand why) :
- The validation in the previous operator worked : I can't see anything specific in the [logs](https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=Osdu_ingest_by_reference&task_id=validate_manifest_schema_task&execution_date=2024-01-19T18%3A27%3A41.994248%2B00%3A00).
- I tried to change all UUIDs (present in the WPC "id") to see if the problem is really on the "id" value and nothing else. I then produced this new manifest :[test-resqml-geosiris-azure-reidfied.json](/uploads/4628d3de7f54d18e50247730b7dcf298/test-resqml-geosiris-azure-reidfied.json).
And the ingestion worked.
- **Conclusion** : the problem is really on these specific ids (the generated "datasets" are never skipped because their id are generated randomly after a push on the DatasetService)
- When a recordId allready exists in the catalog, the ingestion fail
- I tested to ingest twice an other manifest with WITSML data [test-witsml-geosiris.json](/uploads/52042fc0b5bc46097a9d290b12638361/test-witsml-geosiris.json)). And it worked.
- I tried to modify in the previously ingested WITSML manifest the version number, tags to see if the ingestion did not failed because the input was exactly the same. But even with these small modification it worked.
- **Conclusion** : even if an entity with the same "id" is present in the catalog, the ingestion works.
The only difference I see with my "double ingestion" (that worked) and the first ingestion (that failed), is that in the catalog the current version of the WPC (not mine) contains "DDMS" attributes.
Is there any chance that the DDMS attributes give to WPC a security to avoid WPC replacement and keep the integrity of the DDMS ?
Help :
- Get the current version of the WPC that makes mine fail to be ingested :
POSTMAN GET request : *https://{{STORAGE_HOST}}/records/opendes:work-product-component--LocalBoundaryFeature:5fa99eb4-b11f-4f08-b1be-2d64ff14286f*jean-francois RAINAUDChad LeongDeepanshu Singhaljean-francois RAINAUDhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/684AZURE M22 - Core Services - Augmented Index2024-02-06T13:59:29ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brAZURE M22 - Core Services - Augmented IndexHello,
I'm trying to Reindex the Records but getting response 404 Not Found.
So far what I've node is:
- Collection: [Azure_OSDU_Smoke_Tests.postman_collection.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/ma...Hello,
I'm trying to Reindex the Records but getting response 404 Not Found.
So far what I've node is:
- Collection: [Azure_OSDU_Smoke_Tests.postman_collection.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Azure-M22/Services/Smoke-tests/Azure_OSDU_Smoke_Tests.postman_collection.json?ref_type=heads)
- Environment: [Verify_Preshipping_New_TeamA-E.postman_environment.json
](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Azure-M22/Environment/Verify_Preshipping_New_TeamA-E.postman_environment.json?ref_type=heads)
## 1. Create Recrod Using Storage Service
PUT https://{{STORAGE_HOST}}/records/
**BODY CURL VERSION**
```
curl --location --request PUT 'https://osdu-ship.msft-osdu-test.org/api/storage/v2/records/' \
--header 'data-partition-id: opendes' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <Token>' \
--header 'Content-Type: application/json' \
--data-raw '[
{
"acl": {
"owners": [
"data.default.owners@opendes.contoso.com"
],
"viewers": [
"data.default.viewers@opendes.contoso.com"
]
},
"data": {
"Source": "Juliana",
"PlayID": "01292024-JFA",
"PlayName": "JFA29JAN",
"PlayDescription": "Creating a record do Augmented Index",
"NameAliases": [
{
"AliasName": "SFA-ID01292024",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:RegulatoryIdentifier:"
}
]
},
"id": "opendes:master-data--Play:Play-JFA29JAN",
"kind": "osdu:wks:master-data--Play:1.0.0",
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-4375119"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [],
"version": 0
}
]
'
```
**BODY**
```json
[
{
"acl": {
"owners": [
"{{New_OwnerDataGroup}}@{{data-partition-id}}{{domain}}"
],
"viewers": [
"{{New_ViewerDataGroup}}@{{data-partition-id}}{{domain}}"
]
},
"data": {
"Source": "Juliana",
"PlayID": "01292024-JFA",
"PlayName": "JFA29JAN",
"PlayDescription": "Creating a record do Augmented Index",
"NameAliases": [
{
"AliasName": "SFA-ID01292024",
"AliasNameTypeID": "{{data-partition-id}}:reference-data--AliasNameType:RegulatoryIdentifier:"
}
]
},
"id": "{{data-partition-id}}:master-data--Play:Play-JFA29JAN",
"kind": "osdu:wks:master-data--Play:1.0.0",
"legal": {
"legaltags": [
"{{LegalTagNameExists}}"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [],
"version": 0
}
]
```
**RESPONSE**
```json
{
"recordCount": 1,
"recordIds": [
"opendes:master-data--Play:Play-JFA29JAN"
],
"skippedRecordIds": [],
"recordIdVersions": [
"opendes:master-data--Play:Play-JFA29JAN:1706530251825746"
]
}
```
## 2. Retrieve Record using Storage
GET https://{{STORAGE_HOST}}/records/{{recordID}}
**BODY CURL VERSION**
```
curl --location 'https://osdu-ship.msft-osdu-test.org/api/storage/v2/records/opendes:master-data--Play:Play-JFA29JAN' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer <Token>'
```
**RESPONSE**
```json
{
"data": {
"Source": "Juliana",
"PlayID": "01292024-JFA",
"PlayName": "JFA29JAN",
"PlayDescription": "Creating a record do Augmented Index",
"NameAliases": [
{
"AliasName": "SFA-ID01292024",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:RegulatoryIdentifier:"
}
]
},
"meta": [],
"id": "opendes:master-data--Play:Play-JFA29JAN",
"version": 1706530251825746,
"kind": "osdu:wks:master-data--Play:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-4375119"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping@azureglobal1.onmicrosoft.com",
"createTime": "2024-01-29T12:10:52.772Z"
}
```
## 3. Retrieve Record using Search
POST https://{{SEARCH_HOST}}/query
**BODY CURL VERSION**
```
curl --location 'https://osdu-ship.msft-osdu-test.org/api/search/v2/query' \
--header 'Authorization: Bearer <Token>' \
--header 'data-partition-id: opendes' \
--header 'Content-Type: application/json' \
--data '{
"kind": "osdu:wks:master-data--Play:1.0.0",
"query": "id: \"opendes:master-data--Play:Play-JFA29JAN\" "
}
'
```
**BODY**
```json
{
"kind": "osdu:wks:master-data--Play:1.0.0",
"query": "id: \"opendes:master-data--Play:Play-JFA29JAN\" "
}
```
**RESPONSE**
```json
{
"results": [
{
"data": {
"VirtualProperties.DefaultLocation.QuantitativeAccuracyBandID": null,
"ResourceLifecycleStatus": null,
"ResourceCurationStatus": null,
"TechnicalAssuranceID": null,
"PlayID": "01292024-JFA",
"VirtualProperties.DefaultLocation.SpatialGeometryTypeID": null,
"Source": "Juliana",
"VirtualProperties.DefaultName": "JFA29JAN",
"PlayDescription": "Creating a record do Augmented Index",
"VersionCreationReason": null,
"VirtualProperties.DefaultLocation.CoordinateQualityCheckPerformedBy": null,
"ResourceSecurityClassification": null,
"SpatialLocation.SpatialParameterTypeID": null,
"ExistenceKind": null,
"SpatialLocation.CoordinateQualityCheckPerformedBy": null,
"NameAliases": [
{
"AliasName": "SFA-ID01292024",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:RegulatoryIdentifier:",
"DefinitionOrganisationID": null
}
],
"PlayTypeID": null,
"DCCustomRegulatoryID": "SFA-ID01292024",
"TechnicalAssuranceTypeID": null,
"PlayName": "JFA29JAN",
"SpatialLocation.QualitativeSpatialAccuracyTypeID": null,
"SpatialLocation.SpatialGeometryTypeID": null,
"VirtualProperties.DefaultLocation.SpatialParameterTypeID": null,
"ResourceHomeRegionID": null,
"VirtualProperties.DefaultLocation.QualitativeSpatialAccuracyTypeID": null,
"SpatialLocation.QuantitativeAccuracyBandID": null
},
"kind": "osdu:wks:master-data--Play:1.0.0",
"source": "wks",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"type": "master-data--Play",
"version": 1706530251825746,
"tags": {
"normalizedKind": "osdu:wks:master-data--Play:1"
},
"createTime": "2024-01-29T12:10:52.772Z",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-4375119"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping@azureglobal1.onmicrosoft.com",
"id": "opendes:master-data--Play:Play-JFA29JAN"
}
],
"aggregations": null,
"phraseSuggestions": [],
"totalCount": 1
}
```
## 4. Create Reference data for Augmented Index
PUT https://{{STORAGE_HOST}}/records/
**BODY CURL VERSION**
```
curl --location --request PUT 'https://osdu-ship.msft-osdu-test.org/api/storage/v2/records/' \
--header 'data-partition-id: opendes' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <Token>' \
--header 'Content-Type: application/json' \
--data-raw '[
{
"acl": {
"owners": [
"data.default.owners@opendes.contoso.com"
],
"viewers": [
"data.default.viewers@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-4375119"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [],
"data": {
"Name": "Play-IndexPropertyPathConfiguration",
"Description": "valid for all master-data--Play kinds for major version 1.",
"Code": "osdu:wks:master-data--Play:1.",
"AttributionAuthority": "OSDU",
"Configurations": [
{
"Name": "DCCustomRegulatoryID",
"Policy": "ExtractFirstMatch",
"Paths": [
{
"ValueExtraction": {
"RelatedConditionMatches": [
"opendes:reference-data--AliasNameType:RegulatoryIdentifier:"
],
"RelatedConditionProperty": "data.NameAliases[].AliasNameTypeID",
"ValuePath": "data.NameAliases[].AliasName"
}
}
],
"UseCase": "As a user I want to discover and match Play records by their custom field value."
}
]
},
"id": "opendes:reference-data--IndexPropertyPathConfiguration:wks:master-data--Play:1.",
"kind": "osdu:wks:reference-data--IndexPropertyPathConfiguration:1.0.0",
"version": 0
}
]'
```
**BODY**
```json
[
{
"acl": {
"owners": [
"{{New_OwnerDataGroup}}@{{data-partition-id}}{{domain}}"
],
"viewers": [
"{{New_ViewerDataGroup}}@{{data-partition-id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{LegalTagNameExists}}"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"meta": [],
"data": {
"Name": "Play-IndexPropertyPathConfiguration",
"Description": "valid for all master-data--Play kinds for major version 1.",
"Code": "osdu:wks:master-data--Play:1.",
"AttributionAuthority": "OSDU",
"Configurations": [
{
"Name": "DCCustomRegulatoryID",
"Policy": "ExtractFirstMatch",
"Paths": [
{
"ValueExtraction": {
"RelatedConditionMatches": [
"{{data-partition-id}}:reference-data--AliasNameType:RegulatoryIdentifier:"
],
"RelatedConditionProperty": "data.NameAliases[].AliasNameTypeID",
"ValuePath": "data.NameAliases[].AliasName"
}
}
],
"UseCase": "As a user I want to discover and match Play records by their custom field value."
}
]
},
"id": "{{data-partition-id}}:reference-data--IndexPropertyPathConfiguration:wks:master-data--Play:1.",
"kind": "osdu:wks:reference-data--IndexPropertyPathConfiguration:1.0.0",
"version": 0
}
]
```
**RESPONSE**
```json
{
"recordCount": 1,
"recordIds": [
"opendes:reference-data--IndexPropertyPathConfiguration:wks:master-data--Play:1."
],
"skippedRecordIds": [],
"recordIdVersions": [
"opendes:reference-data--IndexPropertyPathConfiguration:wks:master-data--Play:1.:1706530435891902"
]
}
```
## 5. Reindex Records
POST https://{{INDEXER_HOST}}/reindex?force_clean=true
**BODY CURL VERSION**
```
curl --location 'https://osdu-ship.msft-osdu-test.org/api/indexer/v2/ /reindex?force_clean=true' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer <Token>' \
--data '{
"kind": "osdu:wks:master-data--Play:1.0.0"
}'
```
**BODY**
```json
{
"kind": "osdu:wks:master-data--Play:1.0.0"
}
```
**RESPONSE**
```json
{
"timestamp": "2024-01-29T12:17:35.896+00:00",
"status": 404,
"error": "Not Found",
"path": "/api/indexer/v2/%20/reindex"
}
```
Regards,M22 - Release 0.25https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/660Manifest Ingestion - Schema validation error when persistable reference not p...2024-01-29T15:15:37ZChad LeongManifest Ingestion - Schema validation error when persistable reference not provided## Introduction:
In manifest ingestion, a [validation step](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/blob/master/osdu_ingestion/libs/validation/validate_schema.py?ref_type=heads) is performe...## Introduction:
In manifest ingestion, a [validation step](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/blob/master/osdu_ingestion/libs/validation/validate_schema.py?ref_type=heads) is performed to validate the schema according to the schema in https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Generated/abstract/AbstractMetaItem.1.0.0.json?ref_type=heads#L148 , the required field is `"kind"` and `"persistableReference"`.
This [MR](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/796) has been introduced with [ADR 188](https://community.opengroup.org/osdu/platform/system/storage/-/issues/188) where the UnitofMeasureID can be provided instead of persistable reference.
This results in a schema validation error when persistable reference is not provided. Example payload, in the meta block where the persistableReference is not provided.
```json
{
"executionContext": {
"acl": {
"owners": ["data.default.owners@osdu.example.com"],
"viewers": ["data.default.viewers@osdu.example.com"]
},
"legal": { "legaltags": ["osdu-public-usa-dataset"], "otherRelevantDataCountries": ["US"] },
"Payload": { "AppKey": "test-app", "data-partition-id": "osdu" },
"manifest": {
"kind": "osdu:wks:Manifest:1.0.0",
"MasterData": [
{
"id": "osdu:master-data--Well:Well_CL_883_0",
"data": {
"Source": "TNO",
"NameAliases": [
{
"AliasNameTypeID": "osdu:reference-data--AliasNameType:WELL_NAME:",
"AliasName": "ACA-0"
},
{ "AliasNameTypeID": "osdu:reference-data--AliasNameType:UWI:", "AliasName": "0" }
],
"GeoContexts": [
{
"GeoPoliticalEntityID": "osdu:master-data--GeoPoliticalEntity:Netherlands_Country:",
"GeoTypeID": "osdu:reference-data--GeoPoliticalEntityType:Country:"
},
{
"GeoPoliticalEntityID": "osdu:master-data--GeoPoliticalEntity:Limburg_Province:",
"GeoTypeID": "osdu:reference-data--GeoPoliticalEntityType:State%2FProvinceID:"
},
{
"GeoPoliticalEntityID": "osdu:master-data--GeoPoliticalEntity:L_Quadrant:",
"GeoTypeID": "osdu:reference-data--GeoPoliticalEntityType:Quadrant:"
}
],
"SpatialLocation": {
"Wgs84Coordinates": {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": { "type": "Point", "coordinates": [5.98136045, 51.43503877] },
"properties": {}
}
]
}
},
"FacilityTypeID": "osdu:reference-data--FacilityType:Well:",
"FacilityOperators": [
{ "FacilityOperatorOrganisationID": "osdu:master-data--Organisation:ROVD:" }
],
"VerticalMeasurements": [
{
"VerticalCRSID": "osdu:reference-data--CoordinateReferenceSystem:MSL:",
"VerticalMeasurement": 100,
"VerticalMeasurementID": "Rotary Table",
"VerticalMeasurementPathID": "osdu:reference-data--VerticalMeasurementPath:Elevation:",
"VerticalMeasurementUnitOfMeasureID": "osdu:reference-data--UnitOfMeasure:M:"
}
],
"FacilityID": "ACA-0"
},
"legal": { "legaltags": ["osdu-public-usa-dataset"], "otherRelevantDataCountries": ["US"] },
"kind": "osdu:wks:master-data--Well:1.0.0",
"acl": {
"owners": ["data.default.owners@osdu.example.com"],
"viewers": ["data.default.viewers@osdu.example.com"]
},
"meta": [
{
"kind": "Unit",
"name": "ft",
"unitOfMeasureID": "osdu:reference-data--UnitOfMeasure:ft:",
"propertyNames": ["VerticalMeasurements[].VerticalMeasurement"]
}
]
}
]
}
}
}
```
## Error:
```json
Manifest kind: osdu:wks:master-data--Well:1.0.0
[2024-01-15, 20:17:26 UTC] {validate_schema.py:321} ERROR - Error: {'name': 'ft', 'unitOfMeasureID': 'opendes:reference-data--UnitOfMeasure:ft:', 'kind': 'Unit', 'propertyNames': ['VerticalMeasurements[].VerticalMeasurement']} is not valid under any of the given schemas
Failed validating 'oneOf' in schema['properties']['meta']['items']:
...
```
AWS: https://a452580d-4da3-4e22-aa4c-edcf6286c5c2.c5.us-east-2.airflow.amazonaws.com/log?dag_id=Osdu_ingest&task_id=validate_manifest_schema_task&execution_date=2024-01-15T19%3A48%3A04.766160%2B00%3A00&map_index=-1
Azure: https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=Osdu_ingest&task_id=validate_manifest_schema_task&execution_date=2024-01-15T20%3A17%3A12.621733%2B00%3A00
GC: https://548a04f7b3aa4e52a897b94772864397-dot-us-central1.composer.googleusercontent.com/log?dag_id=Osdu_ingest&task_id=validate_manifest_schema_task&execution_date=2024-01-15T20%3A17%3A22.072263%2B00%3A00
RI: https://airflow.bm22.gcp.gnrg-osdu.projects.epam.com/log?dag_id=Osdu_ingest&task_id=validate_manifest_schema_task&execution_date=2024-01-15T20%3A17%3A26.295718%2B00%3A00https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/628M20 IBM Error in CSV parser2024-01-29T13:15:36ZTaylor GraberM20 IBM Error in CSV parserI am unable to have success through the csv-parser-dag. The log returns these error messages:
[2023-10-17 16:12:26,580] {pod_launcher.py:149} INFO - Error starting ApplicationContext. To display the conditions report re-run your applicat...I am unable to have success through the csv-parser-dag. The log returns these error messages:
[2023-10-17 16:12:26,580] {pod_launcher.py:149} INFO - Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
[2023-10-17 16:12:26,584] {pod_launcher.py:149} INFO - 2023-10-17 16:12:26.583 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
I've attached the full log as well if that is helpful. Please let me know if you need more information from me[FullLog.txt](/uploads/102a9075fb9de58b0189f0b5ec6650a2/FullLog.txt)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/656Logging and Monitoring For IBM (Unable to Access IBM Redhat Console)2024-01-29T06:10:21ZNaufal Mohamed NooriLogging and Monitoring For IBM (Unable to Access IBM Redhat Console)Similar to issue I have in https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/587 we would like to have IBM to open up a console monitor for IBM to be used for logging and monitoring procedure in preship
OR
You have y...Similar to issue I have in https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/587 we would like to have IBM to open up a console monitor for IBM to be used for logging and monitoring procedure in preship
OR
You have your personel to test out based on the existing test case here: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M16/Test_Plan_Results_M16/Operational%20Procedures%20Test/R3M16_IBM_OSDU_PTP_TeamA_Non_Destructive_Operational_Procedure_Test_NN.docx
cc @davidglasshttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/681RI M22 - Core Services - Testing Dataset Collection no uploading the file2024-01-26T16:30:47ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brRI M22 - Core Services - Testing Dataset Collection no uploading the fileHello,
I'm trying to upload a file to complete the task Testing Dataset Collection - Dataset API but I'm getting the response not found..
I need some help to figure out what is going on...
Here is what I`ve done:
- Collection: [RI_OSDU...Hello,
I'm trying to upload a file to complete the task Testing Dataset Collection - Dataset API but I'm getting the response not found..
I need some help to figure out what is going on...
Here is what I`ve done:
- Collection: [RI_OSDU_Smoke_Tests.postman_collection.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/RI-M22/RI_OSDU_Smoke_Tests.postman_collection.json?ref_type=heads)
- Environment: [OSDU_R3_RI_Pre-ship.postman_environment.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/RI-M22/OSDU_R3_RI_Pre-ship.postman_environment.json?ref_type=heads)
## 1. Create LegalTag
POST https://{{LEGAL_HOST}}/legaltags
**BODY CURL VERSION**
```
curl --location 'https://osdu.bm22.gcp.gnrg-osdu.projects.epam.com/api/legal/v1/legaltags' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer <Token>' \
--data '{
"name": "osdu-DatasetAPI-Legal-Tag-Test9872069",
"description": "Legal Tag added for OpenVDS",
"properties": {
"contractId": "123456",
"countryOfOrigin": [
"US",
"CA"
],
"dataType": "Third Party Data",
"exportClassification": "EAR99",
"originator": "Auto Test",
"personalData": "No Personal Data",
"securityClassification": "Private",
"expirationDate": "2025-12-25"
}
}'
```
**BODY**
```json
{
"name": "{{datasetTagName}}",
"description": "Legal Tag added for OpenVDS",
"properties": {
"contractId": "123456",
"countryOfOrigin": [
"US",
"CA"
],
"dataType": "Third Party Data",
"exportClassification": "EAR99",
"originator": "Auto Test",
"personalData": "No Personal Data",
"securityClassification": "Private",
"expirationDate": "2025-12-25"
}
}
```
**RESPONSE**
```json
{
"name": "osdu-DatasetAPI-Legal-Tag-Test9872069",
"description": "Legal Tag added for OpenVDS",
"properties": {
"countryOfOrigin": [
"US",
"CA"
],
"contractId": "123456",
"expirationDate": "2025-12-25",
"originator": "Auto Test",
"dataType": "Third Party Data",
"securityClassification": "Private",
"personalData": "No Personal Data",
"exportClassification": "EAR99"
}
}
```
## 2. Get Storage Instructions
POST {{DATASET_HOST}}/storageInstructions?kindSubType=dataset--FileCollection.Generic
**BODY CURL VERSION**
```
curl --location --request POST 'https://osdu.bm22.gcp.gnrg-osdu.projects.epam.com/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer <Token>'
```
**RESPONSE**
```json
{
"providerKey": "ANTHOS",
"storageLocation": {
"url": "https://s3.bm22.gcp.gnrg-osdu.projects.epam.com/refi-osdu-staging-area/",
"createdBy": "osdu-tester@service.local",
"fileCollectionSource": "4de9dc5d7f3a4d4896f401dfb0447934",
"signingOptions": {
"x-amz-date": "20240125T143252Z",
"x-amz-signature": "a0a39358cba1c663a1a6afabe6ed3c7b181bb28088b5a765e5bce7455ef06f39",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "fileUser/20240125/us-east-1/s3/aws4_request",
"policy": "eyJleHBpcmF0aW9uIjoiMjAyNC0wMS0yNVQxNTozMjo1Mi45ODlaIiwiY29uZGl0aW9ucyI6W1siZXEiLCIkYnVja2V0IiwicmVmaS1vc2R1LXN0YWdpbmctYXJlYSJdLFsic3RhcnRzLXdpdGgiLCIka2V5IiwiNGRlOWRjNWQ3ZjNhNGQ0ODk2ZjQwMWRmYjA0NDc5MzQvIl0sWyJlcSIsIiR4LWFtei1hbGdvcml0aG0iLCJBV1M0LUhNQUMtU0hBMjU2Il0sWyJlcSIsIiR4LWFtei1jcmVkZW50aWFsIiwiZmlsZVVzZXIvMjAyNDAxMjUvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJdLFsiZXEiLCIkeC1hbXotZGF0ZSIsIjIwMjQwMTI1VDE0MzI1MloiXV19"
}
}
}
```
## 3a. Upload file 1 to S3
PUT https://storage.googleapis.com/upload/storage/v1/b/{{unsigned_url}}/o?uploadType=media&name={{folder}}%2Ffile1
**BODY CURL VERSION**
```
curl --location 'https://storage.googleapis.com/upload/storage/v1/b/https://s3.bm22.gcp.gnrg-osdu.projects.epam.com/refi-osdu-staging-area//o?uploadType=media&name=4de9dc5d7f3a4d4896f401dfb0447934%2Ffile1' \
--header 'Content-Type: text/csv' \
--data '@/C:/Users/juliana.fernandes/Documents/OSDU/Preshipping/R3M22/Plataform Validation/Test_02/wellbore.csv'
```
**BODY - binary**
```
wellbore.csv
```
**RESPONSE**
```
Not Found
```
Regards,M22 - Release 0.25Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/675RI M22 - Core Services - Register and Notification2024-01-26T13:46:01ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brRI M22 - Core Services - Register and NotificationHello,
I'm trying to do the task Register and Notification for pre-shipping and I did not found any collection to use at RI folder.
So I used the collection provided by GC and I have a question if I'm using the right pushEndpoint and ...Hello,
I'm trying to do the task Register and Notification for pre-shipping and I did not found any collection to use at RI folder.
So I used the collection provided by GC and I have a question if I'm using the right pushEndpoint and where I can get this pushEndpoint to create a subscription.
About the Notification, there is any documentation showing how to do it?
Here is What I did:
- Collection: [Denis-EPAM-Registration_API_CI-CD_v1.0_subscription.postman_collection.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/GC-M22/Denis-EPAM-Registration_API_CI-CD_v1.0_subscription.postman_collection.json)
- Environment: [OSDU_R3_RI_Pre-ship.postman_environment.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/RI-M22/OSDU_R3_RI_Pre-ship.postman_environment.json?ref_type=heads)
## 1. Create Test Subscription
POST https://{{HOSTNAME}}/api/register/v1/subscription
**BODY CURL VERSION**
```
curl --location 'https://osdu.bm22.gcp.gnrg-osdu.projects.epam.com/api/register/v1/subscription' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <Token>' \
--data '{
"name": "test-subscription",
"description": "test description",
"topic": "records-changed",
"pushEndpoint": "https://preship.gcp.gnrg-osdu.projects.epam.com/api/register/v1/test-gc/challenge/test_subscription_2",
"secret": {
"secretType": "HMAC",
"value": "7a786376626e"
}
}'
```
**BODY**
```json
{
"name": "test-subscription",
"description": "test description",
"topic": "records-changed",
"pushEndpoint": "https://preship.gcp.gnrg-osdu.projects.epam.com/api/register/v1/test-gc/challenge/test_subscription_2",
"secret": {
"secretType": "HMAC",
"value": "7a786376626e"
}
}
```
**RESPONSE**
```
{
"id": "cmVjb3Jkcy1jaGFuZ2VkaHR0cHM6Ly9wcmVzaGlwLmdjcC5nbnJnLW9zZHUucHJvamVjdHMuZXBhbS5jb20vYXBpL3JlZ2lzdGVyL3YxL3Rlc3QtZ2MvY2hhbGxlbmdlL3Rlc3Rfc3Vic2NyaXB0aW9uXzI=",
"name": "test-subscription",
"description": "test description",
"topic": "records-changed",
"pushEndpoint": "https://preship.gcp.gnrg-osdu.projects.epam.com/api/register/v1/test-gc/challenge/test_subscription_2",
"createdBy": "osdu-tester@service.local",
"notificationId": "de-b1f08167-5bb3-47f8-a6ed-0b6edab61d23",
"secret": {
"secretType": "HMAC",
"value": "7a786376626e"
}
}
```
Regards,
JulianaM22 - Release 0.25Denis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/633Error in M21 GC OSDU Smoke tests2024-01-26T08:14:39ZCarl GodkinError in M21 GC OSDU Smoke testsWhile doing some testing, I noticed a mistake (I think) in one of the Google Postman collections called [GC OSDU Smoke Tests](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M21/GC-M21/GC_OSDU_Smoke_Tests.postma...While doing some testing, I noticed a mistake (I think) in one of the Google Postman collections called [GC OSDU Smoke Tests](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M21/GC-M21/GC_OSDU_Smoke_Tests.postman_collection.json).
The tests under Core Services/Search beginning with "A01" and "C01" each have a mistake in the body:
```
"kind": "{{data-partition-id}}:{{schemaSource}}:master-data--Well:*.*.*",
```
For the M21Google pre-ship environment, the `data-partition-id` is `m19` but these tests don't match anything unless you change ``{{data-parition-id}} to `osdu`.
Fixing these to use``{{authority}}`` (I think that's correct) would help future newbies. Thanks.Denis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/680GC M22 - Core Services - Notification2024-01-25T19:42:48ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brGC M22 - Core Services - NotificationHello,
I'm trying to do the Notification task at preshipping but not sure how to get it...
One of the ways is using the request "Check Changes" but I would like to know how to do this at Airflow...
Where should I look for it? Which DA...Hello,
I'm trying to do the Notification task at preshipping but not sure how to get it...
One of the ways is using the request "Check Changes" but I would like to know how to do this at Airflow...
Where should I look for it? Which DAG? And Which ID should I used to look into airflow?
Attached is my test for Register...
Regards,
Julian[M22_GC_Core-Services-Register-and-Notification-Juliana-Fernandes.md](/uploads/2cfd6b5edbcf1ca1d5f7496631dcdc4f/M22_GC_Core-Services-Register-and-Notification-Juliana-Fernandes.md)M22 - Release 0.25Denis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/663SDUTIL not working on Seismic DDMS IBM pre-shipping environment2024-01-24T22:23:56ZBhuwan Prasad UpadhyaySDUTIL not working on Seismic DDMS IBM pre-shipping environmentI am trying to validate the Seismic DDMS IBM pre-shipping environment. Unfortunately, I am not able to create a subproject. I keep getting the following error:
```
> Registering the subproject fgxm22(tenant=opendes) with osdu-bvt as adm...I am trying to validate the Seismic DDMS IBM pre-shipping environment. Unfortunately, I am not able to create a subproject. I keep getting the following error:
```
> Registering the subproject fgxm22(tenant=opendes) with osdu-bvt as admin ...
[500] At least one of `iamProfileName` or `iamProfileId` must be specified.
```
Command that I am using to create subproject:
```bash
python sdutil mk sd://$DATA_PARTITION_ID/$SUBPROJECT $PRESHIPPING_USER $LEGAL_TAG --data_partition_id=$DATA_PARTITION_ID --idtoken=$ACCESS_TOKEN
```
Where I need to define `iamProfileName` or `iamProfileId`?vikas ranavikas rana