Azure - 50K Performance Load Test Fails
Please can you help investigate why 50K load testing is failing for Azure?
Below is an extract of the DAG Log: [2022-03-02 03:33:05,837] {single_manifest_processor.py:76} DEBUG - Process entity SRN: opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_37665 [2022-03-02 03:33:05,837] {process_manifest_r3.py:115} INFO - Sending records to Storage service [2022-03-02 03:33:05,837] {process_manifest_r3.py:116} DEBUG - [{'data': {'OrganisationName': 'Name_TEST_BULK_2022-03-02_132_37665', 'Source': 'Source_TEST_BULK_2022-03-02_132_37665'}, 'kind': 'opendes:wks:master-data--Organisation:1.0.0', 'legal': {'legaltags': ['opendes-public-usa-dataset-open-test-data'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_37665', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}] [2022-03-02 03:33:05,837] {logging_mixin.py:104} INFO - Attribute "ancestry" is missing from dict being converted to record [2022-03-02 03:33:05,839] {connectionpool.py:226} DEBUG - Starting new HTTP connection (1): storage.osdu-azure.svc.cluster.local:80 [2022-03-02 03:33:06,047] {connectionpool.py:433} DEBUG - http://storage.osdu-azure.svc.cluster.local:80 "PUT /api/storage/v2/records HTTP/1.1" 201 None [2022-03-02 03:33:06,050] {process_manifest_r3.py:123} DEBUG - Response: {'recordCount': 1, 'recordIds': ['opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_37665'], 'skippedRecordIds': [], 'recordIdVersions': ['opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_37665:1646191985865838']} [2022-03-02 03:33:06,050] {process_manifest_r3.py:124} INFO - Records 'opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_37665' were saved using Storage service. [2022-03-02 03:33:06,050] {single_manifest_processor.py:76} DEBUG - Process entity SRN: opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_802 [2022-03-02 03:33:06,050] {process_manifest_r3.py:115} INFO - Sending records to Storage service [2022-03-02 03:33:06,050] {process_manifest_r3.py:116} DEBUG - [{'data': {'OrganisationName': 'Name_TEST_BULK_2022-03-02_132_802', 'Source': 'Source_TEST_BULK_2022-03-02_132_802'}, 'kind': 'opendes:wks:master-data--Organisation:1.0.0', 'legal': {'legaltags': ['opendes-public-usa-dataset-open-test-data'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_802', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}] [2022-03-02 03:33:06,051] {logging_mixin.py:104} INFO - Attribute "ancestry" is missing from dict being converted to record [2022-03-02 03:33:06,053] {connectionpool.py:226} DEBUG - Starting new HTTP connection (1): storage.osdu-azure.svc.cluster.local:80 [2022-03-02 03:33:06,262] {connectionpool.py:433} DEBUG - http://storage.osdu-azure.svc.cluster.local:80 "PUT /api/storage/v2/records HTTP/1.1" 201 None [2022-03-02 03:33:06,263] {process_manifest_r3.py:123} DEBUG - Response: {'recordCount': 1, 'recordIds': ['opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_802'], 'skippedRecordIds': [], 'recordIdVersions': ['opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_802:1646191986074393']} [2022-03-02 03:33:06,263] {process_manifest_r3.py:124} INFO - Records 'opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_802' were saved using Storage service. [2022-03-02 03:33:06,263] {single_manifest_processor.py:76} DEBUG - Process entity SRN: opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_19234 [2022-03-02 03:33:06,263] {process_manifest_r3.py:115} INFO - Sending records to Storage service [2022-03-02 03:33:06,263] {process_manifest_r3.py:116} DEBUG - [{'data': {'OrganisationName': 'Name_TEST_BULK_2022-03-02_132_19234', 'Source': 'Source_TEST_BULK_2022-03-02_132_19234'}, 'kind': 'opendes:wks:master-data--Organisation:1.0.0', 'legal': {'legaltags': ['opendes-public-usa-dataset-open-test-data'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_19234', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}] [2022-03-02 03:33:06,263] {logging_mixin.py:104} INFO - Attribute "ancestry" is missing from dict being converted to record [2022-03-02 03:33:06,265] {connectionpool.py:226} DEBUG - Starting new HTTP connection (1): storage.osdu-azure.svc.cluster.local:80 [2022-03-02 03:33:06,493] {connectionpool.py:433} DEBUG - http://storage.osdu-azure.svc.cluster.local:80 "PUT /api/storage/v2/records HTTP/1.1" 201 None [2022-03-02 03:33:06,494] {process_manifest_r3.py:123} DEBUG - Response: {'recordCount': 1, 'recordIds': ['opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_19234'], 'skippedRecordIds': [], 'recordIdVersions': ['opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_19234:1646191986288644']} [2022-03-02 03:33:06,494] {process_manifest_r3.py:124} INFO - Records 'opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_19234' were saved using Storage service. [2022-03-02 03:33:06,494] {single_manifest_processor.py:76} DEBUG - Process entity SRN: opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_35038 [2022-03-02 03:33:06,494] {process_manifest_r3.py:115} INFO - Sending records to Storage service [2022-03-02 03:33:06,495] {process_manifest_r3.py:116} DEBUG - [{'data': {'OrganisationName': 'Name_TEST_BULK_2022-03-02_132_35038', 'Source': 'Source_TEST_BULK_2022-03-02_132_35038'}, 'kind': 'opendes:wks:master-data--Organisation:1.0.0', 'legal': {'legaltags': ['opendes-public-usa-dataset-open-test-data'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:master-data--Organisation:TEST_BULK_2022-03-02_132_35038', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}] [2022-03-02 03:33:06,495] {logging_mixin.py:104} INFO - Attribute "ancestry" is missing from dict being converted to record [2022-03-02 03:33:06,497] {connectionpool.py:226} DEBUG - Starting new HTTP connection (1): storage.osdu-azure.svc.cluster.local:80 [2022-03-02 03:33:06,725] {process_utils.py:113} WARNING - process psutil.Process(pid=4532, name='airflow task ru', status='sleeping', started='00:47:11') did not respond to SIGTERM. Trying SIGKILL [2022-03-02 03:33:06,739] {process_utils.py:66} INFO - Process psutil.Process(pid=4532, name='airflow task ru', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='00:47:11') (4532) terminated with exit code Negsignal.SIGKILL [2022-03-02 03:33:06,740] {standard_task_runner.py:128} ERROR - Job 2439 was killed before it finished (likely due to running out of memory) [2022-03-02 03:33:06,740] {base_job.py:227} DEBUG - [heartbeat]
Details of run from the python consoled is noted below.
CURRENT DAG RUN STATUS: running INFO:root:Wait 10 sec ... check DAG Osdu_ingest for runId: 71bae3df-82f6-42de-a55a-0536006f5760 ... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): osdu-ship.msft-osdu-test.org:443 DEBUG:urllib3.connectionpool:https://osdu-ship.msft-osdu-test.org:443 "GET /api/workflow/v1/workflow/Osdu_ingest/workflowRun/71bae3df-82f6-42de-a55a-0536006f5760 HTTP/1.1" 401 14 DEBUG:root:HTTP GET https://osdu-ship.msft-osdu-test.org/api/workflow/v1/workflow/Osdu_ingest/workflowRun/71bae3df-82f6-42de-a55a-0536006f5760 DEBUG:root:args = {"headers": {"data-partition-id": "opendes", "Accept": "application/json", "Authorization": "Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ik1yNS1BVWliZkJpaTdOZDFqQmViYXhib1hXMCIsImtpZCI6Ik1yNS1BVWliZkJpaTdOZDFqQmViYXhib1hXMCJ9.eyJhdWQiOiJhYjMyMGVkMy05Y2RkLTQ3OTgtOGUzYy0yYTY1NzgwMDE4M2IiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC81ODk3NWZkMy00OTc3LTQ0ZDAtYmVhOC0zN2FmMGJhYWMxMDAvIiwiaWF0IjoxNjQ2MTc4MzAyLCJuYmYiOjE2NDYxNzgzMDIsImV4cCI6MTY0NjE4MzMwNSwiYWNyIjoiMSIsImFpbyI6IkFUUUF5LzhUQUFBQUE0aFBxTUpPQmFxRUxwWlIrZEZ0ZDl0REtzRnAyYzJGT3gwVzdSWlRSb2w1NmJ2Y3dXeWZVQnBqS3BhUEhJWVQiLCJhbXIiOlsicHdkIl0sImFwcGlkIjoiYWIzMjBlZDMtOWNkZC00Nzk4LThlM2MtMmE2NTc4MDAxODNiIiwiYXBwaWRhY3IiOiIxIiwiZmFtaWx5X25hbWUiOiJEZWZhdWx0IiwiZ2l2ZW5fbmFtZSI6IlByZXNoaXAiLCJpcGFkZHIiOiIxNzEuNzYuMTU0LjEzNSIsIm5hbWUiOiJQcmVzaGlwcGluZyIsIm9pZCI6IjhlMGI0NjQ0LTBlZDAtNDFmMi04YzZkLWQ2YWFhNzk4NjE4MiIsInJoIjoiMC5BVGNBMDEtWFdIZEowRVMtcURldkM2ckJBTk1PTXF2ZG5KaEhqandxWlhnQUdEczNBTVUuIiwic2NwIjoiVXNlci5SZWFkIiwic3ViIjoiOTdwUWdKdFJGSDk5WTFLVml3RlY0R2FBRHhLc0llUkc5WlBKLTRQbk1iMCIsInRpZCI6IjU4OTc1ZmQzLTQ5NzctNDRkMC1iZWE4LTM3YWYwYmFhYzEwMCIsInVuaXF1ZV9uYW1lIjoicHJlc2hpcHBpbmdAYXp1cmVnbG9iYWwxLm9ubWljcm9zb2Z0LmNvbSIsInVwbiI6InByZXNoaXBwaW5nQGF6dXJlZ2xvYmFsMS5vbm1pY3Jvc29mdC5jb20iLCJ1dGkiOiJLb2c1TmtoRWQwR0JRUXFGNGI0TEFBIiwidmVyIjoiMS4wIn0.yenAh5rOmXIPb6KsB0tqI9gAG_sLBVQjB3cw_X6uF00C6BjGh9S_8_0FYBNRkKpf-mtnTToWoaxWdxWK4NG6rvyJJf654Wsh1cKpIEV09R2tk1U6TBVGjmYdz3lH3E7t-i-PNNYP7RGoXnY3fkoKe4k8Dz5OiC02Yzw2cQt017HWjZ8pO_smKpAKWlvGPSsA1eWmLuqsJsKRv4hGMG8PHAC-EPrhreKmM6U1gm9yX9_JEgg47H5S8MBXzU_AFTOIxMcKHIZPTkAjiR_8uJk_Bh6EVrzlPtDddvc7hZDDHClZbO3eCW7MZBBREt0fYwx2Ttb70opiYlf2G5qist3ryw"}, "verify": false} DEBUG:root:Response: 401 DEBUG:root:text = Jwt is expired <osdu_client.OsduClient object at 0x000001865E669D80>