M20 AWS Airflow Missing Skipped IDs and Logs Level Error
There are 2 issues surrounding current preship AWS airflow. When user performs manifest ingestion with expected failure (i.e. missing reference value):
- The airflow log XCom part does not show proper skipped_ids table as we saw in older milestone release
In R3M16 version, we can see the XCom table has 3 rows including column header:
- The log does not show any ERROR logging level as expected I have checked any stage of the airflow log I dont see any ERROR type logging eventhough the ingestion is failing (due to reference list I purposely include in the JSON manifest which does not exists in the database).
Check runID: be401335-e916-4e14-a38b-0feb52ca1b97
Sampel json used:
"data": {
"ProjectName": "KALICO 3D",
"SpatialLocation": {
"QualitativeSpatialAccuracyTypeID": "{{data_partition_id}}:reference-data--QualitativeSpatialAccuracyType:Assumed_HAHAHA:",
...
...
...}