csv-parser issueshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues2023-05-24T15:35:07Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/81While performing the same workflow using the same data to ingest reference da...2023-05-24T15:35:07ZKamlesh TodaiWhile performing the same workflow using the same data to ingest reference data type, I see that the new record ids are not getting created each time, Instead the version is getting incrementedWhen, try to ingest data where entity type is master, I see the new record id getting created each time I run the workflow, even though the collection and data file being used are same.
To me this is the expected behavior.
When I try to...When, try to ingest data where entity type is master, I see the new record id getting created each time I run the workflow, even though the collection and data file being used are same.
To me this is the expected behavior.
When I try to do the same with entity type reference, I am seeing that the ids getting generated are same (not new) and the only new version is getting generated.
So e.g. if I get count of the records for the entity type before and after the ingestion, the count remains same except for the first time. (in my case the count goes up by 4 for the first time and then it stays the same as my data has 4 records).
`So I modified the file to have one more record (total 5). When I ran the workflow again with additional one record, I saw the count going up by 1 and not by 5.
Before (inserting 5 records)
{
"results": [
{
"id": "opendes:reference-data--ContractorType:LineClearing"
}
],
"aggregations": [
{
"key": "osdu:wks:reference-data--ContractorType:1.0.0",
"count": 9
}
],
"totalCount": 9
}
After (inserting 5 records)
{
"results": [
{
"id": "opendes:reference-data--ContractorType:LineClearing"
}
],
"aggregations": [
{
"key": "osdu:wks:reference-data--ContractorType:1.0.0",
"count": 10
}
],
"totalCount": 10
}
Before (again inserting 5 records)
{
"results": [
{
"id": "opendes:reference-data--ContractorType:LineClearing"
}
],
"aggregations": [
{
"key": "osdu:wks:reference-data--ContractorType:1.0.0",
"count": 10
}
],
"totalCount": 10
}
After (again inserting 5 records)
{
"results": [
{
"id": "opendes:reference-data--ContractorType:LineClearing"
}
],
"aggregations": [
{
"key": "osdu:wks:reference-data--ContractorType:1.0.0",
"count": 10
}
],
"totalCount": 10
}
`
[CSVWorkflow__CI-CD_v2.0-ReferenceData.postman_collection.json](/uploads/3d0652f4e89be166a525d7ff18731c2d/CSVWorkflow__CI-CD_v2.0-ReferenceData.postman_collection.json)
[ReferenceData.csv](/uploads/a04f12cbeaacb300c4d23788032518ae/ReferenceData.csv) (with f5 records)
The environment file can be gotten from
https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M16/QA_Artifacts_M16/envFilesAndCollections/envFiles
OR
https://community.opengroup.org/osdu/platform/testing/-/tree/master/Postman%20Collection/00_CICD_Setup_Environment
@tdixon @debasisc @chadM18 - Release 0.21https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/74CSV Collection in Platform Validation project is not working in AWS environme...2022-04-22T18:13:09ZKamlesh TodaiCSV Collection in Platform Validation project is not working in AWS environment. Works in other environments.The collection CSVWorkflow__CI-CD_v1.0.postman_collection.json is not working in the AWS environment. It works in all other environments.
The difference is that for AWS environment DATASET APIs are used to upload CSV file and for others ...The collection CSVWorkflow__CI-CD_v1.0.postman_collection.json is not working in the AWS environment. It works in all other environments.
The difference is that for AWS environment DATASET APIs are used to upload CSV file and for others FileAPIs are used.Okoun-Ola Fabien HouetoOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/59E2E Tests for csv parser - AWS2022-08-24T14:43:36ZChris ZhangE2E Tests for csv parser - AWSThis is to track the AWS team's work for Integration E2E Tests for CSV Parser.
Related to issue 42 https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/42This is to track the AWS team's work for Integration E2E Tests for CSV Parser.
Related to issue 42 https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/42M10 - Release 0.13GregGreghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/53CSV Parser -R3M7/AWS - issue loading Reference data (Basin Type)2021-12-04T00:35:57ZDebasis ChatterjeeCSV Parser -R3M7/AWS - issue loading Reference data (Basin Type)Originally reported by Kenneth in Preship site
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/212
The problem occurs for import Basin Type Master Data by CSV Ingestion.
See log from CSV_Parser
[C...Originally reported by Kenneth in Preship site
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/212
The problem occurs for import Basin Type Master Data by CSV Ingestion.
See log from CSV_Parser
[CSVParserErrorLog.txt](/uploads/4811e152d00736bd93813814878a9fef/CSVParserErrorLog.txt)