csv-parser issueshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues2023-09-01T14:05:35Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/82Update postman collection path for CSV parsers IT2023-09-01T14:05:35ZChad LeongUpdate postman collection path for CSV parsers ITFailure in https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/390 shows that the postman collection path needs to be updated to reflect the latest test directory and collections in ht...Failure in https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/390 shows that the postman collection path needs to be updated to reflect the latest test directory and collections in https://community.opengroup.org/osdu/qa/-/tree/main/Postman%20Collection/31_CICD_Setup_CSVIngestion
| CSP | Change path to |
| ------ | ------ |
| AWS | https://community.opengroup.org/osdu/qa/-/blob/main/Postman%20Collection/31_CICD_Setup_CSVIngestion/CSVWorkflow_AWS_CI-CD_v2.0..postman_collection_NotRun.json |
| IBM | https://community.opengroup.org/osdu/qa/-/blob/main/Postman%20Collection/31_CICD_Setup_CSVIngestion/CSVWorkflow_CI-CD_v2.0.postman_collection.json |
| GC | https://community.opengroup.org/osdu/qa/-/blob/main/Postman%20Collection/31_CICD_Setup_CSVIngestion/CSVWorkflow_CI-CD_v2.0.postman_collection.json |
- [X] Azure - Done
- [X] AWS - https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/merge_requests/391
- [ ] IBM - Pending
- [ ] GC - PendingM20 - Release 0.23https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/81While performing the same workflow using the same data to ingest reference da...2023-05-24T15:35:07ZKamlesh TodaiWhile performing the same workflow using the same data to ingest reference data type, I see that the new record ids are not getting created each time, Instead the version is getting incrementedWhen, try to ingest data where entity type is master, I see the new record id getting created each time I run the workflow, even though the collection and data file being used are same.
To me this is the expected behavior.
When I try to...When, try to ingest data where entity type is master, I see the new record id getting created each time I run the workflow, even though the collection and data file being used are same.
To me this is the expected behavior.
When I try to do the same with entity type reference, I am seeing that the ids getting generated are same (not new) and the only new version is getting generated.
So e.g. if I get count of the records for the entity type before and after the ingestion, the count remains same except for the first time. (in my case the count goes up by 4 for the first time and then it stays the same as my data has 4 records).
`So I modified the file to have one more record (total 5). When I ran the workflow again with additional one record, I saw the count going up by 1 and not by 5.
Before (inserting 5 records)
{
"results": [
{
"id": "opendes:reference-data--ContractorType:LineClearing"
}
],
"aggregations": [
{
"key": "osdu:wks:reference-data--ContractorType:1.0.0",
"count": 9
}
],
"totalCount": 9
}
After (inserting 5 records)
{
"results": [
{
"id": "opendes:reference-data--ContractorType:LineClearing"
}
],
"aggregations": [
{
"key": "osdu:wks:reference-data--ContractorType:1.0.0",
"count": 10
}
],
"totalCount": 10
}
Before (again inserting 5 records)
{
"results": [
{
"id": "opendes:reference-data--ContractorType:LineClearing"
}
],
"aggregations": [
{
"key": "osdu:wks:reference-data--ContractorType:1.0.0",
"count": 10
}
],
"totalCount": 10
}
After (again inserting 5 records)
{
"results": [
{
"id": "opendes:reference-data--ContractorType:LineClearing"
}
],
"aggregations": [
{
"key": "osdu:wks:reference-data--ContractorType:1.0.0",
"count": 10
}
],
"totalCount": 10
}
`
[CSVWorkflow__CI-CD_v2.0-ReferenceData.postman_collection.json](/uploads/3d0652f4e89be166a525d7ff18731c2d/CSVWorkflow__CI-CD_v2.0-ReferenceData.postman_collection.json)
[ReferenceData.csv](/uploads/a04f12cbeaacb300c4d23788032518ae/ReferenceData.csv) (with f5 records)
The environment file can be gotten from
https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M16/QA_Artifacts_M16/envFilesAndCollections/envFiles
OR
https://community.opengroup.org/osdu/platform/testing/-/tree/master/Postman%20Collection/00_CICD_Setup_Environment
@tdixon @debasisc @chadM18 - Release 0.21https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/79Error diagnostics - need to improve significantly2022-12-13T00:31:21ZDebasis ChatterjeeError diagnostics - need to improve significantlyYou may start of by checking here.
https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M14/AWS-M14/Ingestion%20DAG%20CSV
For each and every problem, I did not get suitable clue from error log.
1. problem in data. ...You may start of by checking here.
https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M14/AWS-M14/Ingestion%20DAG%20CSV
For each and every problem, I did not get suitable clue from error log.
1. problem in data. ELEVATION has non numeric value.
2. problem in schema - TVD, Latitude, Longitude - missed "type=string".
3. At times when the file is missed (incorrect sequence in collection), it gives fatal error instead of saying clearly that "Unable to get the CSV file".
Caused situation where record gets created, we can see all properties from Storage service, but none from Search service.
Nearly impossible to figure out, for average Data Loader (user).
Next, imagine we are ingesting 1000 rows from source CSV and problem occurs in row-253 and row-455.
User's expectation is that CSV Ingestion program should pinpoint and clearly indicate row number and type of problem which caused the failure.
cc @chad , @tdixonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/74CSV Collection in Platform Validation project is not working in AWS environme...2022-04-22T18:13:09ZKamlesh TodaiCSV Collection in Platform Validation project is not working in AWS environment. Works in other environments.The collection CSVWorkflow__CI-CD_v1.0.postman_collection.json is not working in the AWS environment. It works in all other environments.
The difference is that for AWS environment DATASET APIs are used to upload CSV file and for others ...The collection CSVWorkflow__CI-CD_v1.0.postman_collection.json is not working in the AWS environment. It works in all other environments.
The difference is that for AWS environment DATASET APIs are used to upload CSV file and for others FileAPIs are used.Okoun-Ola Fabien HouetoOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/59E2E Tests for csv parser - AWS2022-08-24T14:43:36ZChris ZhangE2E Tests for csv parser - AWSThis is to track the AWS team's work for Integration E2E Tests for CSV Parser.
Related to issue 42 https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/42This is to track the AWS team's work for Integration E2E Tests for CSV Parser.
Related to issue 42 https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/42M10 - Release 0.13GregGreghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/53CSV Parser -R3M7/AWS - issue loading Reference data (Basin Type)2021-12-04T00:35:57ZDebasis ChatterjeeCSV Parser -R3M7/AWS - issue loading Reference data (Basin Type)Originally reported by Kenneth in Preship site
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/212
The problem occurs for import Basin Type Master Data by CSV Ingestion.
See log from CSV_Parser
[C...Originally reported by Kenneth in Preship site
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/212
The problem occurs for import Basin Type Master Data by CSV Ingestion.
See log from CSV_Parser
[CSVParserErrorLog.txt](/uploads/4811e152d00736bd93813814878a9fef/CSVParserErrorLog.txt)