In GCP R3 M10 environments cannot run CSV workflow successfully.
While running CSV workflow in dev2 (QA/PV) environment I am seeing he following in CSV DAG airflow log [2022-03-16 21:30:01,862] {taskinstance.py:1465} ERROR - Task failed with exception Traceback (most recent call last): File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1166, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task result = task_copy.execute(context=context) File "/opt/python3.8/lib/python3.8/site-packages/osdu_airflow/operators/update_status.py", line 123, in execute status_updater = UpdateStatus( File "/opt/python3.8/lib/python3.8/site-packages/osdu_ingestion/libs/update_status.py", line 76, in init config_manager=DefaultConfigManager() File "/opt/python3.8/lib/python3.8/site-packages/osdu_api/configuration/config_manager.py", line 45, in init self._parser = self._read_config(config_file_path) File "/opt/python3.8/lib/python3.8/site-packages/osdu_api/configuration/config_manager.py", line 96, in _read_config raise configparser.Error(f"Could not find the config file in '{config_file_path}'.") configparser.Error: Could not find the config file in 'osdu_api.ini'. [2022-03-16 21:30:01,866] {taskinstance.py:1511} INFO - Marking task as FAILED. dag_id=csv_ingestion, task_id=update_status_running, execution_date=20220316T212453, start_date=20220316T213000, end_date=20220316T213001
{ "workflowId": "4e630551-0521-40d9-ad87-0dcd693d2c50", "runId": "test_workflow305794", "startTimeStamp": 1647465892384, "endTimeStamp": 1647467058633, "status": "failed", "submittedBy": "kamlesh_todai@osdu-gcp.go3-nrg.projects.epam.com" }
I can use the csv workflow collection in pre-ship environment and it returns finished status { "workflowId": "d8d61ca7-3b6e-4c18-b9c4-f10512028bb1", "runId": "test_workflow947857", "startTimeStamp": 1647467499340, "endTimeStamp": 1647467556111, "status": "finished", "submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com" } The issue over there is then I cannot find the inserted records using the Search API, but using the Storage API I can retrieve the inserted records.