Store manifest_by_reference Run Details in Activity template
As a Developer,
I want to store run details (such as failed records, ingested records, and failure information, etc) within the activity template, so that I can retrieve this information later for analysis without relying on the Airflow logger. This is similar like handling osdu_ingest run details in EDS.
Acceptance Criteria: The system should store details of each run, including:
- Number of failed records.
- Number of successfully ingested records.
- Detailed information about any failures (error messages, exceptions, etc.).
- These details should be saved in the activity template associated with the runid.
- The activity record should be easily retrievable for future analysis and troubleshooting.
- No dependency on Airflow's logger for accessing the stored run details.
- The stored activity information should be accessible via a query or reporting tool to facilitate analysis.
Notes:
- The data stored in the activity template should be structured and easy to query.
- This feature should enable post-run analysis without the need to dig into Airflow logs.
Edited by Jeyakumar Devarajulu