Error diagnostics - need to improve significantly
You may start of by checking here. https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M14/AWS-M14/Ingestion%20DAG%20CSV
For each and every problem, I did not get suitable clue from error log.
- problem in data. ELEVATION has non numeric value.
- problem in schema - TVD, Latitude, Longitude - missed "type=string".
- At times when the file is missed (incorrect sequence in collection), it gives fatal error instead of saying clearly that "Unable to get the CSV file".
Caused situation where record gets created, we can see all properties from Storage service, but none from Search service.
Nearly impossible to figure out, for average Data Loader (user).
Next, imagine we are ingesting 1000 rows from source CSV and problem occurs in row-253 and row-455. User's expectation is that CSV Ingestion program should pinpoint and clearly indicate row number and type of problem which caused the failure.