Azure M12 Load Testing 50K records fails due to memory
Just over 40K records are ingested before log file in Airflow reports error as below:
[2022-07-04, 14:05:22 UTC] {logging_mixin.py:109} INFO - Attribute "ancestry" is missing from dict being converted to record [2022-07-04, 14:05:22 UTC] {connectionpool.py:228} DEBUG - Starting new HTTP connection (1): storage.osdu-azure.svc.cluster.local:80 [2022-07-04, 14:05:22 UTC] {process_utils.py:138} WARNING - process psutil.Process(pid=776, name='airflow task ru', status='sleeping', started='11:08:13') did not respond to SIGTERM. Trying SIGKILL [2022-07-04, 14:05:22 UTC] {process_utils.py:75} INFO - Sending the signal Signals.SIGKILL to group 776 [2022-07-04, 14:05:22 UTC] {process_utils.py:70} INFO - Process psutil.Process(pid=776, name='airflow task ru', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='11:08:13') (776) terminated with exit code Negsignal.SIGKILL [2022-07-04, 14:05:22 UTC] {standard_task_runner.py:133} ERROR - Job 24800 was killed before it finished (likely due to running out of memory) [2022-07-04, 14:05:22 UTC] {base_job.py:228} DEBUG - [heartbeat]
Can you please assist to resolve?