[GCP Airflow] XCOM Skipped_Ids Is Not Working
When ingestion record failed and no record being saved we dont see any entry record in XCOM skipped_ids.
Steps to reproduce:
a) Using DAG manifest ingestion, load a master data wellbore record. Note that this record will definitely failed due to missing WellId reference in the record. Hence at the end we expect this record json wont be saved:
BODY:
{
"executionContext": {
"Payload": {
"AppKey": "test-app",
"data-partition-id": "{{data-partition-id}}"
},
"manifest": {
"kind": "{{data-partition-id}}:wks:Manifest:1.0.0",
"MasterData": [
{
"id": "{{data-partition-id}}:master-data--Wellbore:Test_NN_2021_09_24_01",
"kind": "{{data-partition-id}}:wks:master-data--Wellbore:1.0.0",
"acl": {
"owners": [
"data.default.owners@{{data-partition-id}}.osdu-gcp.go3-nrg.projects.epam.com"
],
"viewers": [
"data.default.viewers@{{data-partition-id}}.osdu-gcp.go3-nrg.projects.epam.com"
]
},
"legal": {
"legaltags": [
"{{data-partition-id}}-demo-legaltag"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"WellID": "{{data-partition-id}}:master-data--Well:TEST_ERROR:",
"FacilityName": "TEST_NN_1_ALIAS",
"SequenceNumber": 1,
"Source": "TEST_NN_1_ALIAS_SOURCE",
"NameAliases": [
{
"AliasName": "TEST_NN_1_ALIAS"
}
]
}
}
]
}
}
}
b) Run DAG Manifest POST: https://{{WORKFLOW_HOST}}/workflow/Osdu_ingest/workflowRun:
{ "workflowId": "ef82cba0-0e45-4df3-91bf-4df1553102d3", "runId": "5a786c6f-103e-44d3-b192-d34e3026b722", "startTimeStamp": 1632812342734, "status": "submitted", "submittedBy": "[preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com](mailto:preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com)" }
c) Observe the airflow log. In XCOM no skipped_ids are recorded even the log says no record being saved:
[2021-09-28 07:00:05,878] {process_manifest_r3.py:167} DEBUG - Manifest data: {'ReferenceData': [], 'MasterData': [], 'Data': {'Datasets': [], 'WorkProductComponents': [], 'WorkProduct': {}}, 'kind': 'odesprod:wks:Manifest:1.0.0'}
[2021-09-28 07:00:05,879] {manifest_analyzer.py:286} DEBUG - Entity graph {}.
[2021-09-28 07:00:05,881] {single_manifest_processor.py:136} INFO - Processed ids []
[2021-09-28 07:00:05,882] {process_manifest_r3.py:173} INFO - Processed ids []
(TESTED ON R3M8 Preship GCP environment on 27 September 2021)