EDS XCOM summary Improvement

Below are the suggestion is to change the presentation to make more meaningful for the tester/user.

Change suggestion –

  1. Change “Reference error” to “Referential integrity error”.
  2. Change logic of displaying value in “Schema validation” and “Legal tag/ACL validation” to be the same like “reference error”. Means – when there is no error, it is empty.
  3. When indeed something is wrong in “schema validation”, then modify display to have prefix of “Dag-run-ID=197e3…” instead of simply saying “197e3” .
  4. Another thought is to show two more lines – one for “Reference value mapping used?” and another for “Parent data mapping used?”.
  5. Can this be the very first line? Do we have control of sequence? “Manifest Dag Run ID” and change text to “Manifest Ingestion Dag Run ID”.

Overall, I like the improvement a lot and Xcom summary is going to be more helpful for tester/user to figure out what exactly happened in the “ETL journey”.

Below is the sorted XCOM summary

  1. Manifest Dag Run ID - Osdu_ingest=run_id Can be pushed as first xcom item
  2. Count - Total Fetched Records - To include total no. of fetched records
  3. Count - Successfully Ingested records - Successfully Ingested record(s) count
  4. Count - Failed records - Failed record count
  5. Id list - Successfully Ingested record(s)
  6. Id list - Failed record(s)
  7. Error - Schema Validation
  8. Error - Referential Integrity
  9. Error - Legal Tag - ACL Validation- if no issues, keep it empty - []
  10. Naturalization - eds_naturalization=run_id
Edited by Jeyakumar Devarajulu