AWS Performance Loading Fails
@Wibben @jhogwill @fhoueto.amz
I have tried running performance load testing on AWS but it keep failing. It fails 500; 50 & even 10 records; all on the same phase on "process_single_manifest_file_task" with similar error messages. Below FYI
#500 (closed) Records aws_client.manifest_ingest('test-app', 'AWS/output/osdu_master_data_Organisation_recs500_2022-02-28_16-58-45.json').wait_for_workflow() runId: 9997473f-10ed-4078-ac7c-52c64a076609 Passed on provide_manifest_integrity_task Failed during "process_single_manifest_file_task
*** Failed to verify remote log exists s3://osr3m10-ingest-logbucket-1ipw-s3airflowbucketprod-1hqkljv7hf677/logs/Osdu_ingest/process_single_manifest_file_task/2022-02-28T17:10:10.138035+00:00/1.log. An error occurred (403) when calling the HeadObject operation: Forbidden *** Falling back to local log *** Log file does not exist: /opt/airflow/logs/Osdu_ingest/process_single_manifest_file_task/2022-02-28T17:10:10.138035+00:00/1.log *** Fetching from: http://airflow-worker-6bdcbd95c4-zp4qd:8793/log/Osdu_ingest/process_single_manifest_file_task/2022-02-28T17:10:10.138035+00:00/1.log *** Failed to fetch log file from worker. [Errno -2] Name or service not known
Second Run 500 Records _"runId": "e42608a3-5ab5-42ef-99a9-24fb9c941d0d", _ Passed on provide_manifest_integrity_task Failed during "process_single_manifest_file_task
Error on "process_single_manifest_file_task" *** Failed to verify remote log exists s3://osr3m10-ingest-logbucket-1ipw-s3airflowbucketprod-1hqkljv7hf677/logs/Osdu_ingest/process_single_manifest_file_task/2022-02-28T17:32:43.220746+00:00/1.log. An error occurred (403) when calling the HeadObject operation: Forbidden *** Falling back to local log *** Log file does not exist: /opt/airflow/logs/Osdu_ingest/process_single_manifest_file_task/2022-02-28T17:32:43.220746+00:00/1.log *** Fetching from: http://airflow-worker-6bdcbd95c4-zp4qd:8793/log/Osdu_ingest/process_single_manifest_file_task/2022-02-28T17:32:43.220746+00:00/1.log *** Failed to fetch log file from worker. [Errno -2] Name or service not known
Third Run: : 10 Records "runId": "3aefa10b-5785-46ec-a876-38fb4877a4b9" Passed on provide_manifest_integrity_task Failed during "process_single_manifest_file_task
Error on "process_single_manifest_file_task" *** Failed to verify remote log exists s3://osr3m10-ingest-logbucket-1ipw-s3airflowbucketprod-1hqkljv7hf677/logs/Osdu_ingest/process_single_manifest_file_task/2022-02-28T17:36:13.705214+00:00/1.log. An error occurred (403) when calling the HeadObject operation: Forbidden *** Falling back to local log *** Log file does not exist: /opt/airflow/logs/Osdu_ingest/process_single_manifest_file_task/2022-02-28T17:36:13.705214+00:00/1.log *** Fetching from: http://airflow-worker-6bdcbd95c4-zp4qd:8793/log/Osdu_ingest/process_single_manifest_file_task/2022-02-28T17:36:13.705214+00:00/1.log *** Failed to fetch log file from worker. [Errno -2] Name or service not known