CSV Ingestion - Horizon 3 - Workflow Developer Tasks
CSV Ingestion : Horizon 3
These tasks will be community delivered:
Validation of processing multiple CSV files concurrently via multiple dags.
Automatic schema validation and creation.
Ability to identify states for the jobs. Success, Failed and Succeeded with Warnings, Succeeded with Failures.
Ability to create and support job counters : Successfully records, failed records, bad records and differentiate between data maturity states (raw, WKS, WKE, etc).