AWS R3M9 - Manifest-based Ingestion - Referential Integrity, no clues in Airflow log for failed cases (non-existent Reference value)
Please see details here. AWS_Manifest-based-Ingestion-Referential-Integrity.docx
I introduced bad data on purpose as test case. No clue in provide_manifest_integrity_task log for reason of failure. No clue in XCom summary of process_single_manifest_file_task component.
The behaviour seems different than previous versions.
For Data Loader, it becomes impossible to troubleshoot, correct the error and redo the ingestion.
It seems there is similar problem in GCZ Dev environment too.
Please check and advise.
runID = "runId": "f8d5a31d-3196-4ef0-95af-e8c77bfab3d6",
JSON payload is in attached file.
Can you check and advise?
Thank you
cc - @Wibben , @esmira.rafigayeva , @kennethliew