csv-parser issueshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues2023-09-01T14:05:35Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/82Update postman collection path for CSV parsers IT2023-09-01T14:05:35ZChad LeongUpdate postman collection path for CSV parsers ITFailure in https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/390 shows that the postman collection path needs to be updated to reflect the latest test directory and collections in ht...Failure in https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/390 shows that the postman collection path needs to be updated to reflect the latest test directory and collections in https://community.opengroup.org/osdu/qa/-/tree/main/Postman%20Collection/31_CICD_Setup_CSVIngestion
| CSP | Change path to |
| ------ | ------ |
| AWS | https://community.opengroup.org/osdu/qa/-/blob/main/Postman%20Collection/31_CICD_Setup_CSVIngestion/CSVWorkflow_AWS_CI-CD_v2.0..postman_collection_NotRun.json |
| IBM | https://community.opengroup.org/osdu/qa/-/blob/main/Postman%20Collection/31_CICD_Setup_CSVIngestion/CSVWorkflow_CI-CD_v2.0.postman_collection.json |
| GC | https://community.opengroup.org/osdu/qa/-/blob/main/Postman%20Collection/31_CICD_Setup_CSVIngestion/CSVWorkflow_CI-CD_v2.0.postman_collection.json |
- [X] Azure - Done
- [X] AWS - https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/391
- [ ] IBM - Pending
- [ ] GC - PendingM20 - Release 0.23https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/79Error diagnostics - need to improve significantly2022-12-13T00:31:21ZDebasis ChatterjeeError diagnostics - need to improve significantlyYou may start of by checking here.
https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M14/AWS-M14/Ingestion%20DAG%20CSV
For each and every problem, I did not get suitable clue from error log.
1. problem in data. ...You may start of by checking here.
https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M14/AWS-M14/Ingestion%20DAG%20CSV
For each and every problem, I did not get suitable clue from error log.
1. problem in data. ELEVATION has non numeric value.
2. problem in schema - TVD, Latitude, Longitude - missed "type=string".
3. At times when the file is missed (incorrect sequence in collection), it gives fatal error instead of saying clearly that "Unable to get the CSV file".
Caused situation where record gets created, we can see all properties from Storage service, but none from Search service.
Nearly impossible to figure out, for average Data Loader (user).
Next, imagine we are ingesting 1000 rows from source CSV and problem occurs in row-253 and row-455.
User's expectation is that CSV Ingestion program should pinpoint and clearly indicate row number and type of problem which caused the failure.
cc @chad , @tdixon