Azure R3M9 - CSV Ingestion - workflow using custom schema - problems
Used Postman collection as provided by CSP.
Had to fix several things (endpoint/URL), provide missing environment variables. csv_workflow_name, csv_parsing_dag_name Used value of csv-parser-0.12.0 (Hope this is correct)
Did steps 1, 2, 7, 9, 10, 13, 14. Cannot see anything obvious in Airflow log either for successful creation of records or for any error. Step-14 also does not show any records.
Steps here Azure-CSV-custom-steps.txt
Source CSV file (test data) wellbore-DC.csv
Airflow log here Azure-CSV-custom-Airflow-log.txt
Also note that there seems many redundant steps. For schema creation, it is good to use Schema service and there is no need to use deprecated Storage service feature. Also there seems some overlap between FILE service and Dataset service for file handling. Happy to discuss in a call. @manishk , @krganesan
cc - @sehuboy , @esmira.rafigayeva and @tony_lelm for information