CSV Ingestion - Horizon 3 - Workflow Developer Tasks
CSV Ingestion : Horizon 3
These tasks will be community delivered:
Workflow Execution
-
Validation of processing multiple CSV files concurrently via multiple dags. -
Automatic schema validation and creation. -
Ability to identify states for the jobs. Success, Failed and Succeeded with Warnings, Succeeded with Failures. -
Ability to create and support job counters : Successfully records, failed records, bad records and differentiate between data maturity states (raw, WKS, WKE, etc).