GCP 50K Performance Loading Fails
Please can you help investigate why 50K performance loading is failing for GCP?
DAG Run ID for further investigation is fdcdf45b-76be-4060-80f5-3d6e0023acd6
I have attached a sample of the 1K record which passed OK. Same code was used to generate the 50K manifest records.odesprod_master_data_Organisation_recs1000_2022-02-28_14-46-22.json
Below is the interface error observed towards the end.
CURRENT DAG RUN STATUS: running INFO:root:Wait 10 sec ... check DAG Osdu_ingest for runId: fdcdf45b-76be-4060-80f5-3d6e0023acd6 ... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): preship.gcp.gnrg-osdu.projects.epam.com:443 DEBUG:urllib3.connectionpool:https://preship.gcp.gnrg-osdu.projects.epam.com:443 "GET /api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6 HTTP/1.1" 200 None DEBUG:root:HTTP GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6 DEBUG:root:args = {"headers": {"data-partition-id": "odesprod", "Accept": "application/json", "Authorization": "Bearer ya29.A0ARrdaM-4rHRs0iYoZQ852a9pS69xDmdao1JF3L-xCO5W_m7FSGLGnB4yKcIiUXRDeVfGJSLsDkjZwBd-gEQMjo_oH8WapBkECE2rgPjHq7bMjA27sjU5Qe4Wzy0gzzbXWU29KguWhQhl0eEVMSqN_bCamLBfxQ"}, "verify": false} DEBUG:root:Response: 200 DEBUG:root:json = {"workflowId": "ef82cba0-0e45-4df3-91bf-4df1553102d3", "runId": "fdcdf45b-76be-4060-80f5-3d6e0023acd6", "startTimeStamp": 1646253900184, "status": "running", "submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"} CURRENT DAG RUN STATUS: running INFO:root:Wait 10 sec ... check DAG Osdu_ingest for runId: fdcdf45b-76be-4060-80f5-3d6e0023acd6 ... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): preship.gcp.gnrg-osdu.projects.epam.com:443 DEBUG:urllib3.connectionpool:https://preship.gcp.gnrg-osdu.projects.epam.com:443 "GET /api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6 HTTP/1.1" 200 None DEBUG:root:HTTP GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6 DEBUG:root:args = {"headers": {"data-partition-id": "odesprod", "Accept": "application/json", "Authorization": "Bearer ya29.A0ARrdaM-4rHRs0iYoZQ852a9pS69xDmdao1JF3L-xCO5W_m7FSGLGnB4yKcIiUXRDeVfGJSLsDkjZwBd-gEQMjo_oH8WapBkECE2rgPjHq7bMjA27sjU5Qe4Wzy0gzzbXWU29KguWhQhl0eEVMSqN_bCamLBfxQ"}, "verify": false} DEBUG:root:Response: 200 DEBUG:root:json = {"workflowId": "ef82cba0-0e45-4df3-91bf-4df1553102d3", "runId": "fdcdf45b-76be-4060-80f5-3d6e0023acd6", "startTimeStamp": 1646253900184, "status": "running", "submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"} CURRENT DAG RUN STATUS: running INFO:root:Wait 10 sec ... check DAG Osdu_ingest for runId: fdcdf45b-76be-4060-80f5-3d6e0023acd6 ... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): preship.gcp.gnrg-osdu.projects.epam.com:443 DEBUG:urllib3.connectionpool:https://preship.gcp.gnrg-osdu.projects.epam.com:443 "GET /api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6 HTTP/1.1" 200 None DEBUG:root:HTTP GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6 DEBUG:root:args = {"headers": {"data-partition-id": "odesprod", "Accept": "application/json", "Authorization": "Bearer ya29.A0ARrdaM-4rHRs0iYoZQ852a9pS69xDmdao1JF3L-xCO5W_m7FSGLGnB4yKcIiUXRDeVfGJSLsDkjZwBd-gEQMjo_oH8WapBkECE2rgPjHq7bMjA27sjU5Qe4Wzy0gzzbXWU29KguWhQhl0eEVMSqN_bCamLBfxQ"}, "verify": false} DEBUG:root:Response: 200 DEBUG:root:json = {"workflowId": "ef82cba0-0e45-4df3-91bf-4df1553102d3", "runId": "fdcdf45b-76be-4060-80f5-3d6e0023acd6", "startTimeStamp": 1646253900184, "endTimeStamp": 1646254465687, "status": "failed", "submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"} CURRENT DAG RUN STATUS: failed <osdu_client.OsduClient object at 0x0000022932DCA470>
odesprod_master_data_Organisation_recs1000_2022-02-28_14-46-22.json