Store manifest_by_reference Run Details in Activity template

As a Developer,

I want to store run details (such as failed records, ingested records, and failure information, etc) within the activity template, so that I can retrieve this information later for analysis without relying on the Airflow logger. This is similar like handling osdu_ingest run details in EDS.

Acceptance Criteria: The system should store details of each run, including:

  • Number of failed records.
  • Number of successfully ingested records.
  • Detailed information about any failures (error messages, exceptions, etc.).
  • These details should be saved in the activity template associated with the runid.
  • The activity record should be easily retrievable for future analysis and troubleshooting.
  • No dependency on Airflow's logger for accessing the stored run details.
  • The stored activity information should be accessible via a query or reporting tool to facilitate analysis.

Notes:

  • The data stored in the activity template should be structured and easy to query.
  • This feature should enable post-run analysis without the need to dig into Airflow logs.
Edited by Jeyakumar Devarajulu