AZURE - Unit Conversion - Search Record Failling
Hello Everyone,
Please, I'm trying to do the workflow Unit Conversion and everything works as expected, except the final step when I need to get the record using search services to check if the convertion worked. Maybe I'm missing something. Can I get some help? This is what I'm doing:
M18 AZURE Advanced Search - Juliana
15 June 2023
-
Collection used Unit Conversion: R3_Full_manifest-based_ingestion.postman_collection.json (https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M18/Azure-M18/Services/Ingestion/ManifestIngestion)
-
Environment: Verify_Preshipping_New_TeamA-E.postman_environment.json (https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M18/Azure-M18/Environment)
UNIT CONVERSION
WP Ingestion File Collection
POST https://{{WORKFLOW_URL}}/workflow/Osdu_ingest/workflowRun
curl --location 'https://osdu-ship.msft-osdu-test.org/api/workflow/v1/workflow/Osdu_ingest/workflowRun' \
Body
{
"runId": "{{$guid}}",
"executionContext": {
"Payload": {
"AppKey": "test-app",
"data-partition-id": "{{data-partition-id}}"
},
"manifest": {
"kind": "{{data-partition-id}}:wks:Manifest:1.0.0",
"ReferenceData": [
{
"id": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED",
"kind": "{{authority}}:wks:reference-data--ResourceSecurityClassification:1.0.0",
"data": {
"Source": "Volve Loading",
"Name": "RESTRICTED"
},
"acl": {
"owners": [
"data.default.owners@{{data-partition-id}}{{domain}}"
],
"viewers": [
"data.default.viewers@{{data-partition-id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{LegalTagNameExists}}"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
}
}
],
"MasterData": [
{
"id": "{{data-partition-id}}:master-data--SeismicAcquisitionSurvey:ST0202R08_UC_JFA_Jun152023",
"kind": "{{authority}}:wks:master-data--SeismicAcquisitionSurvey:1.0.0",
"meta": [
{
"kind": "Unit",
"name": "ms",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.001,\"c\":1.0,\"d\":0.0},\"symbol\":\"ms\",\"baseMeasurement\":{\"ancestry\":\"T\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"unitOfMeasureID": "{{data-partition-id}}:reference-data--UnitOfMeasure:ms:",
"propertyNames": [
"RecordLength"
]
},
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.3048,\"c\":1.0,\"d\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"L\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"unitOfMeasureID": "{{data-partition-id}}:reference-data--UnitOfMeasure:ft:",
"propertyNames": [
"ShotpointIncrementDistance",
"CableLength",
"CableSpacingDistance",
"MaxOffsetDistance"
]
}
],
"data": {
"Purpose": "Acquisition for Volve",
"SeismicGeometryTypeID": "{{data-partition-id}}:reference-data--SeismicGeometryType:3D:",
"OperatingEnvironmentID": "{{data-partition-id}}:reference-data--OperatingEnvironment:Offshore:",
"ShotpointIncrementDistance": 25.0,
"EnergySourceTypeID": "{{data-partition-id}}:reference-data--SeismicEnergySourceType:Airgun:",
"SourceArrayCount": 2,
"SourceArraySeparationDistance": 100.0,
"SampleInterval": 4.0,
"RecordLength": 10200,
"CableCount": 4,
"CableLength": 6000.0,
"CableSpacingDistance": 100.0,
"MaxOffsetDistance": 6000.0,
"FoldCount": 120,
"VesselNames": [
"GECO ANGLER"
]
},
"acl": {
"owners": [
"data.default.owners@{{data-partition-id}}{{domain}}"
],
"viewers": [
"data.default.viewers@{{data-partition-id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{LegalTagNameExists}}"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
}
}
]
}
}
}
Result
{
"workflowId": "Osdu_ingest",
"runId": "5bafaa5d-5a76-4a2e-8e21-b0ae92265c8e",
"startTimeStamp": 1686835035501,
"status": "submitted",
"submittedBy": "preshipping@azureglobal1.onmicrosoft.com"
}
Workflow - Get Work Status
GET https://{{WORKFLOW_URL}}/workflow/Osdu_ingest/workflowRun/{{run_id}}
curl --location 'https://osdu-ship.msft-osdu-test.org/api/workflow/v1/workflow/Osdu_ingest/workflowRun/5bafaa5d-5a76-4a2e-8e21-b0ae92265c8e' \
Result
{
"workflowId": "Osdu_ingest",
"runId": "5bafaa5d-5a76-4a2e-8e21-b0ae92265c8e",
"startTimeStamp": 1686835035501,
"endTimeStamp": 1686835094286,
"status": "finished",
"submittedBy": "preshipping@azureglobal1.onmicrosoft.com"
}
Check Airflow - XCom Summary
- Access Airflow: https://osdu-ship.msft-osdu-test.org/airflow2/home
- Access DAG “Osdu_ingest”
- Change the view from “Tree” to “Graph”
- Check the runID the you got on postman
- Click on the graph on “Process_single_manifest_file_task”
- A window will pop up, click on “log” on the top
- Change from “log” to “XCom"
Body
record_ids ['opendes:reference-data--ResourceSecurityClassification:RESTRICTED', 'opendes:master-data--SeismicAcquisitionSurvey:ST0202R08_UC_JFA_Jun152023']
skipped_ids []
Retrieve the Record using STORAGE Service
curl --location 'https://osdu-ship.msft-osdu-test.org/api/storage/v2/records/opendes:master-data--SeismicAcquisitionSurvey:ST0202R08_UC_JFA_Jun152023' \
--header 'data-partition-id: opendes' \
Result
{
"data": {
"CableCount": 4,
"CableLength": 6000,
"CableSpacingDistance": 100,
"EnergySourceTypeID": "opendes:reference-data--SeismicEnergySourceType:Airgun:",
"FoldCount": 120,
"MaxOffsetDistance": 6000,
"OperatingEnvironmentID": "opendes:reference-data--OperatingEnvironment:Offshore:",
"Purpose": "Acquisition for Volve",
"RecordLength": 10200,
"SampleInterval": 4,
"SeismicGeometryTypeID": "opendes:reference-data--SeismicGeometryType:3D:",
"ShotpointIncrementDistance": 25,
"SourceArrayCount": 2,
"SourceArraySeparationDistance": 100,
"VesselNames": [
"GECO ANGLER"
]
},
"meta": [
{
"kind": "Unit",
"name": "ms",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.001,\"c\":1.0,\"d\":0.0},\"symbol\":\"ms\",\"baseMeasurement\":{\"ancestry\":\"T\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"propertyNames": [
"RecordLength"
],
"unitOfMeasureID": "opendes:reference-data--UnitOfMeasure:ms:"
},
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.3048,\"c\":1.0,\"d\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"L\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"propertyNames": [
"ShotpointIncrementDistance",
"CableLength",
"CableSpacingDistance",
"MaxOffsetDistance"
],
"unitOfMeasureID": "opendes:reference-data--UnitOfMeasure:ft:"
}
],
"modifyUser": "preshipping@azureglobal1.onmicrosoft.com",
"modifyTime": "2023-06-15T13:18:09.324Z",
"id": "opendes:master-data--SeismicAcquisitionSurvey:ST0202R08_UC_JFA_Jun152023",
"version": 1686835089118123,
"kind": "osdu:wks:master-data--SeismicAcquisitionSurvey:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-5247622"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping@azureglobal1.onmicrosoft.com",
"createTime": "2023-06-15T13:18:03.831Z"
}
Search Records WP
POST https://{{SEARCH_HOST}}/query
curl --location 'https://osdu-ship.msft-osdu-test.org/api/search/v2/query' \
Body
{
"kind": "{{authority}}:{{schemaSource}}:master-data--SeismicAcquisitionSurvey:1.0.0",
"query": "id:\"{{data-partition-id}}:master-data--SeismicAcquisitionSurvey:ST0202R08_UC_JFA_Jun152023\""
}
Result
{
"results": [],
"aggregations": [],
"totalCount": 0
}