Manifest ingestion (Osdu_ingest) of 50000 records is failing when trying to perform batch_upload. It works fine when batch_upload is not performed and the regular option is used.
In R3M11 pre-ship environment, While executing the workflow to do performance load testing using batch_upload - Manifest ingestion (Osdu_ingest) of 50000 records is failing when trying to perform batch_upload. It works fine when batch_upload is not performed and the regular option (Single process) is used. Gateway is timing out. Here is the console log ue 26-Apr-2022 12:41:11 INFO Selected Cloud Service Provider: ibm Tue 26-Apr-2022 12:41:11 DEBUG Starting new HTTPS connection (1): keycloak-osdu-keycloak.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud:443 Tue 26-Apr-2022 12:41:11 DEBUG https://keycloak-osdu-keycloak.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud:443 "POST /auth/realms/OSDU/protocol/openid-connect/token HTTP/1.1" 200 3501 Tue 26-Apr-2022 12:41:11 DEBUG ### Inserting file json <<<<<<<<<<<: D:\OSDU\PreShipping\M11\loadTesting\IngestionBulkBatch\json\ibm\ibm_batchManifest_2022-04-26_12-14-44_50000.json Tue 26-Apr-2022 12:41:13 DEBUG Starting new HTTPS connection (1): osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud:443 Tue 26-Apr-2022 12:42:27 DEBUG https://osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud:443 "POST /osdu-workflow/api/workflow/v1/workflow/Osdu_ingest/workflowRun HTTP/1.1" 504 164 Tue 26-Apr-2022 12:42:27 DEBUG HTTP POST https://osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/osdu-workflow/api/workflow/v1/workflow/Osdu_ingest/workflowRun Tue 26-Apr-2022 12:42:27 DEBUG Response: 504 Tue 26-Apr-2022 12:42:27 DEBUG text =
**504 Gateway Time-out**504 Gateway Time-out
openresty Tue 26-Apr-2022 12:42:27 ERROR ### HTML 504 with response None Tue 26-Apr-2022 12:42:27 INFO ######### Exiting Process due to error to POST manifest #########
This is the airflow log
*** Log file does not exist: /opt/airflow/logs/Osdu_ingest/update_status_finished_task/2022-04-26T17:41:52.705910+00:00/1.log *** Fetching from: http://airflow-worker-0.airflow-worker.osdu-airflow.svc.cluster.local:8793/log/Osdu_ingest/update_status_finished_task/2022-04-26T17:41:52.705910+00:00/1.log
[2022-04-26 17:43:08,018] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: Osdu_ingest.update_status_finished_task 2022-04-26T17:41:52.705910+00:00 [queued]> [2022-04-26 17:43:08,049] {taskinstance.py:896} INFO - Dependencies all met for <TaskInstance: Osdu_ingest.update_status_finished_task 2022-04-26T17:41:52.705910+00:00 [queued]> [2022-04-26 17:43:08,049] {taskinstance.py:1087} INFO -
[2022-04-26 17:43:08,049] {taskinstance.py:1088} INFO - Starting attempt 1 of 1 [2022-04-26 17:43:08,050] {taskinstance.py:1089} INFO -
[2022-04-26 17:43:08,082] {taskinstance.py:1107} INFO - Executing <Task(UpdateStatusOperator): update_status_finished_task> on 2022-04-26T17:41:52.705910+00:00 [2022-04-26 17:43:08,094] {standard_task_runner.py:52} INFO - Started process 1308 to run task [2022-04-26 17:43:08,108] {standard_task_runner.py:76} INFO - Running: ['', 'tasks', 'run', 'Osdu_ingest', 'update_status_finished_task', '2022-04-26T17:41:52.705910+00:00', '--job-id', '15807', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/osdu-ingest-r3.py', '--cfg-path', '/tmp/tmpzwokqu_w', '--error-file', '/tmp/tmpmglxfmpm'] [2022-04-26 17:43:08,110] {standard_task_runner.py:77} INFO - Job 15807: Subtask update_status_finished_task [2022-04-26 17:43:11,063] {logging_mixin.py:104} INFO - Running <TaskInstance: Osdu_ingest.update_status_finished_task 2022-04-26T17:41:52.705910+00:00 [running]> on host -worker-0.-worker.osdu-.svc.cluster.local [2022-04-26 17:43:13,860] {taskinstance.py:1300} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=*** AIRFLOW_CTX_DAG_ID=Osdu_ingest AIRFLOW_CTX_TASK_ID=update_status_finished_task AIRFLOW_CTX_EXECUTION_DATE=2022-04-26T17:41:52.705910+00:00 AIRFLOW_CTX_DAG_RUN_ID=b90defff-6b20-4b9b-8c0b-aa9d93a90aaf [2022-04-26 17:43:17,232] {update_status.py:66} INFO - There are failed tasks before this one. So it has status FAILED [2022-04-26 17:43:22,353] {logging_mixin.py:104} INFO - env_vars_enabled ************************ true [2022-04-26 17:43:22,353] {logging_mixin.py:104} INFO - cloud provider ******************* ibm [2022-04-26 17:43:22,353] {logging_mixin.py:104} INFO - Inside if [2022-04-26 17:43:26,773] {taskinstance.py:1501} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task result = task_copy.execute(context=context) File "/home/airflow/.local/lib/python3.8/site-packages/osdu_airflow/operators/update_status.py", line 140, in execute raise PipelineFailedError("Dag failed") osdu_ingestion.libs.exceptions.PipelineFailedError: Dag failed [2022-04-26 17:43:26,777] {taskinstance.py:1544} INFO - Marking task as FAILED. dag_id=Osdu_ingest, task_id=update_status_finished_task, execution_date=20220426T174152, start_date=20220426T174308, end_date=20220426T174326 [2022-04-26 17:43:26,919] {local_task_job.py:151} INFO - Task exited with return code 1