AWS - Unable to ingest manifests using Osdu_ingest dag airflow
When I attempt to bulk ingest manifest files using the Osdu_ingest airflow I am getting an exception.
I am using the following scripts to ingest manifest files: https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/Workflow%20KT%20Demos/Performance_Loading_Tests.
The Osdu_ingest workflow does run, however, an error happens during the "process_manifest_task_#" task.
The tasks are marked as a success however the logs indicate that nothing was ingested. Below is the full logs returned for the tasks.
*** Falling back to local log
*** Log file does not exist: /opt/airflow/logs/Osdu_ingest/process_manifest_task_1/2022-03-03T14:38:56.503427+00:00/1.log
*** Fetching from: http://airflow-worker-d5685f99-4s6tv:8793/log/Osdu_ingest/process_manifest_task_1/2022-03-03T14:38:56.503427+00:00/1.log
*** Failed to fetch log file from worker. [Errno -2] Name or service not known
I have also verified via postman that none of the manifests were ingested.
Edited by Michael