CSV Ingestion - Horizon 1 - Workflow Developer Tasks
CSV Ingestion : Horizon 1
These tasks will be performed by the Pune development organizations
Workflow Execution
-
Ability to parse a CSV file. -
Validate the structure of CSV file against a configured CSV schema schema.Column Header Validation, Data validation based on n rows.Schema Service integration. -
Ability of Character Set Conversion to UTF8 -
Ability to create\modify the logic for datarecord id. -
Type Coercion: Typecast integers, doubles, floats, etc as described in the schema to “number” -
Type Coercion: Typecast Degrees/Minutes/Seconds (DMS) or Easting/Northing should be typecast to “String” -
Type Coercion : Typecast dates as described in the schema to a date, doing a date format conversion to ISO8601TZ format (for fully qualified dates). -
Type Coercion: Some date fragments (such as years) cannot be easily converted to this format and should be typecast to a number instead -
Type Coercion: All other encountered attributes should be typecast as string. -
Build JSON payload for batch processing(Identifying success,failed and error records) -
Logging appropriate\segregated application logs - Audit,Error,Warning. -
Sink to Storage - Integration with Storage service. -
Client for Core Services- For dev environment. - Storage,Search etc