AWS Segy to ZGY workflow not working
While attempting to convert a segy dataset to a zgy dataset in seismic ddms using the SEGY_TO_ZGY dag, the taks failed. Here are the workflow task logs.
*** Reading remote log from s3://prer3m8-ingest-logbucket-59uj-s3airflowbucketprod-udbcrvficay5/logs/SEGY_TO_ZGY/segy-to-zgy/2021-09-28T13:45:15.858388+00:00/1.log.
[2021-09-28 13:45:21,816] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: SEGY_TO_ZGY.segy-to-zgy 2021-09-28T13:45:15.858388+00:00 [queued]>
[2021-09-28 13:45:21,840] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: SEGY_TO_ZGY.segy-to-zgy 2021-09-28T13:45:15.858388+00:00 [queued]>
[2021-09-28 13:45:21,840] {taskinstance.py:880} INFO -
--------------------------------------------------------------------------------
[2021-09-28 13:45:21,840] {taskinstance.py:881} INFO - Starting attempt 1 of 1
[2021-09-28 13:45:21,840] {taskinstance.py:882} INFO -
--------------------------------------------------------------------------------
[2021-09-28 13:45:21,858] {taskinstance.py:901} INFO - Executing <Task(KubernetesPodOperator): segy-to-zgy> on 2021-09-28T13:45:15.858388+00:00
[2021-09-28 13:45:21,861] {standard_task_runner.py:54} INFO - Started process 470 to run task
[2021-09-28 13:45:21,892] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'SEGY_TO_ZGY', 'segy-to-zgy', '2021-09-28T13:45:15.858388+00:00', '--job_id', '5128', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/openzgy/segy_to_zgy_ingestion_dag.py', '--cfg_path', '/tmp/tmpxqzm08nt']
[2021-09-28 13:45:21,893] {standard_task_runner.py:78} INFO - Job 5128: Subtask segy-to-zgy
[2021-09-28 13:45:21,956] {logging_mixin.py:120} INFO - Running <TaskInstance: SEGY_TO_ZGY.segy-to-zgy 2021-09-28T13:45:15.858388+00:00 [running]> on host airflow-worker-0.airflow-worker.osdu-airflow.svc.cluster.local
[2021-09-28 13:45:22,028] {logging_mixin.py:120} WARNING - /home/airflow/.local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py:331: DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated. Please use `k8s.V1Pod`.
security_context=_extract_security_context(pod.spec.security_context)
[2021-09-28 13:45:22,028] {logging_mixin.py:120} WARNING - /home/airflow/.local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py:77: DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated. Please use `k8s.V1Pod` instead.
pod = self._mutate_pod_backcompat(pod)
[2021-09-28 13:45:22,078] {pod_launcher.py:173} INFO - Event: segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a had an event of type Pending
[2021-09-28 13:45:22,078] {pod_launcher.py:139} WARNING - Pod not yet started: segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a
[2021-09-28 13:45:23,087] {pod_launcher.py:173} INFO - Event: segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a had an event of type Pending
[2021-09-28 13:45:23,087] {pod_launcher.py:139} WARNING - Pod not yet started: segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a
[2021-09-28 13:45:24,096] {pod_launcher.py:173} INFO - Event: segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a had an event of type Failed
[2021-09-28 13:45:24,097] {pod_launcher.py:284} INFO - Event with job id segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a Failed
[2021-09-28 13:45:24,116] {pod_launcher.py:156} INFO - b'[0.003890] SEGYTOZGY_ZFP_LOD_COMPRESS=[]\n'
[2021-09-28 13:45:24,116] {pod_launcher.py:156} INFO - b'[0.003896] SEGYTOZGY_ZFP_LOD_SNR=[]\n'
[2021-09-28 13:45:24,117] {pod_launcher.py:156} INFO - b'[0.003903] set SEGYTOZGY_INSECURE_PRINT_TOKEN=1 to print token values\n'
[2021-09-28 13:45:24,117] {pod_launcher.py:156} INFO - b'[0.003912] END Environment variables\n'
[2021-09-28 13:45:24,117] {pod_launcher.py:156} INFO - b'[0.003921] Command line arguments: [/usr/local/bin/segy/SegyToZgy] [--osdu] [osdu:dataset--FileCollection.SEGY:e1d8444c4ae545c1b3446211be7995bb] [{{SeismicTraceDataBinGridWPId}}]\n'
[2021-09-28 13:45:24,117] {pod_launcher.py:156} INFO - b'[0.003933] Fetching work product [{{SeismicTraceDataBinGridWPId}}]\n'
[2021-09-28 13:45:24,117] {pod_launcher.py:156} INFO - b'[0.003949] About to get record [{{SeismicTraceDataBinGridWPId}}]\n'
[2021-09-28 13:45:24,117] {pod_launcher.py:156} INFO - b'[0.004017] Storage service URL: [https://preshiptesting.osdu.aws/api/storage/v2]\n'
[2021-09-28 13:45:24,117] {pod_launcher.py:156} INFO - b'[0.004026] Data partition ID : [osdu]\n'
[2021-09-28 13:45:24,117] {pod_launcher.py:156} INFO - b'Invalid format of object reference.\n'
[2021-09-28 13:45:24,132] {pod_launcher.py:173} INFO - Event: segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a had an event of type Failed
[2021-09-28 13:45:24,132] {pod_launcher.py:284} INFO - Event with job id segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a Failed
[2021-09-28 13:45:24,138] {pod_launcher.py:173} INFO - Event: segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a had an event of type Failed
[2021-09-28 13:45:24,138] {pod_launcher.py:284} INFO - Event with job id segy-to-zgy-a6019acd8c6641a587114d9efa5cfa6a Failed
[2021-09-28 13:45:24,172] {taskinstance.py:1150} ERROR - Pod Launching failed: Pod returned a failure: failed
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 309, in execute
'Pod returned a failure: {state}'.format(state=final_state))
airflow.exceptions.AirflowException: Pod returned a failure: failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 979, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 312, in execute
raise AirflowException('Pod Launching failed: {error}'.format(error=ex))
airflow.exceptions.AirflowException: Pod Launching failed: Pod returned a failure: failed
[2021-09-28 13:45:24,173] {taskinstance.py:1194} INFO - Marking task as FAILED. dag_id=SEGY_TO_ZGY, task_id=segy-to-zgy, execution_date=20210928T134515, start_date=20210928T134521, end_date=20210928T134524
[2021-09-28 13:45:26,726] {local_task_job.py:102} INFO - Task exited with return code 1
I have attached the steps I used to test this feature in a file attached to this ticket.