IBM M22 - Unit Convervion - Problems when the Master Data is Triggered
Hello,
I was trying to complete my task FoR Unit Conversion for Manifest Ingest and got an problem when I triggered the Master Data. The ID expected at Airflow is not showing up. Here is what I've done:
- Collection used: https://community.opengroup.org/osdu/qa/-/blob/main/Postman%20Collection/29_CICD_Setup_Ingestion/Manifest_Based_Ingestion_Osdu_ingest_CI-CD_v2.0.postman_collection.json?ref_type=heads
- Environment used:https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/GC-M22/OSDU_R3_GC_Pre-ship.postman_environment.json
1. Create Legal Tag
2. Ingestion Master and Refedence Data
POST {{WORKFLOW_HOST}}/workflow/Osdu_ingest/workflowRun
Body Curl Version
curl --location 'https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-workflow/api/workflow/v1/workflow/Osdu_ingest/workflowRun' \
--header 'data-partition-id: opendes' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <Token>' \
--header 'Cookie: 5e7c4a992d2558462d1c9fcde6f1cee7=4ec498d778b559cf74a987cfbc06b21b' \
--data-raw '{
"runId": "3bd48b11-e720-4284-892b-1a632c61799e",
"executionContext": {
"Payload": {
"AppKey": "test-app",
"data-partition-id": "opendes"
},
"manifest": {
"kind": "osdu:wks:Manifest:1.0.0",
"ReferenceData": [
{
"id": "opendes:reference-data--ResourceSecurityClassification:RESTRICTED",
"kind": "osdu:wks:reference-data--ResourceSecurityClassification:1.0.0",
"data": {
"Source": "Volve Loading",
"Name": "RESTRICTED"
},
"acl": {
"owners": [
"data.default.owners@opendes.ibm.com"
],
"viewers": [
"data.default.viewers@opendes.ibm.com"
]
},
"legal": {
"legaltags": [
"opendes-ManifestDag-Legal-Tag-3669675"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
}
}
],
"MasterData": [
{
"id": "opendes:master-data--SeismicAcquisitionSurvey:ST0202R08-233296",
"kind": "osdu:wks:master-data--SeismicAcquisitionSurvey:1.0.0",
"meta": [
{
"kind": "Unit",
"name": "ms",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.001,\"c\":1.0,\"d\":0.0},\"symbol\":\"ms\",\"baseMeasurement\":{\"ancestry\":\"T\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"unitOfMeasureID": "opendes:reference-data--UnitOfMeasure:ms:",
"propertyNames": [
"RecordLength"
]
},
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.3048,\"c\":1.0,\"d\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"L\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"unitOfMeasureID": "opendes:reference-data--UnitOfMeasure:ft:",
"propertyNames": [
"ShotpointIncrementDistance",
"CableLength",
"CableSpacingDistance",
"MaxOffsetDistance"
]
}
],
"data": {
"Purpose": "Acquisition for Volve",
"SeismicGeometryTypeID": "opendes:reference-data--SeismicGeometryType:3D:",
"OperatingEnvironmentID": "opendes:reference-data--OperatingEnvironment:Offshore:",
"ShotpointIncrementDistance": 25.0,
"EnergySourceTypeID": "opendes:reference-data--SeismicEnergySourceType:Airgun:",
"SourceArrayCount": 2,
"SourceArraySeparationDistance": 100.0,
"SampleInterval": 4.0,
"RecordLength": 10200,
"CableCount": 4,
"CableLength": 6000.0,
"CableSpacingDistance": 100.0,
"MaxOffsetDistance": 6000.0,
"FoldCount": 120,
"VesselNames": [
"GECO ANGLER"
]
},
"acl": {
"owners": [
"data.default.owners@opendes.ibm.com"
],
"viewers": [
"data.default.viewers@opendes.ibm.com"
]
},
"legal": {
"legaltags": [
"opendes-ManifestDag-Legal-Tag-3669675"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
}
}
]
}
}
}'
Response
{
"workflowId": "9bc5ee62ca5c498db13453ec36b2922e",
"runId": "9c748348-0c82-404f-b5ab-178892b37d6c",
"startTimeStamp": 1705468720080,
"status": "submitted",
"submittedBy": "osdu-bvt@osdu.opengroup.org"
}
3. Workflow - Get Workflow Status
GET {{WORKFLOW_HOST}}/workflow/Osdu_ingest/workflowRun/9c748348-0c82-404f-b5ab-178892b37d6c
Body Curl Version
curl --location 'https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-workflow/api/workflow/v1/workflow/Osdu_ingest/workflowRun/9c748348-0c82-404f-b5ab-178892b37d6c' \
--header 'Data-Partition-Id: opendes' \
--header 'Authorization: Bearer <Token>' \
--header 'Cookie: 5e7c4a992d2558462d1c9fcde6f1cee7=4ec498d778b559cf74a987cfbc06b21b' \
--data ''
Response
{
"runId": "9c748348-0c82-404f-b5ab-178892b37d6c",
"startTimeStamp": 1705468720080,
"endTimeStamp": 1705468734017,
"status": "finished",
"submittedBy": "osdu-bvt@osdu.opengroup.org"
}
4.Xcom Summary
- Access Airflow
- Access DAG “Osdu_ingest”
- Change the view from “Tree” to “Graph”
- Check the runID the you got on postman
- Click on the graph on “Process_single_manifest_file_task”
- A window will pop up, click on “log” on the top
- Change from “log” to “XCom"
record_ids ['opendes:reference-data--ResourceSecurityClassification:RESTRICTED']
skipped_ids []
What can be causing this?
Regards,
Juliana Fernandes