Skip to content

RequestBodyTooLarge exception in EDS ingest for large payload.

RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit

 

Few Details

 

Search Url: query_with_cursor

 

Airflow Logs

 

  1. QUERY BODY:

 

Formatted Request (truncated the conditions. I wanted to use IDs in the query, but as reported before EDS is not working with IDs):

 

{

        "kind": "osdu:wks:master-data--Organisation:1.0.0",

        "query": "Name:("TGS" OR "AGIP%20PETROLEUM"............)",

        "sort": {

                       "field": [

                                      "createTime"

                       ],

                       "order": [

                                      "ASC"

                       ]

        },

        "limit": 2

}

 

 

  1. Filter criteria after: Name:("TGS" OR "AGIP%20PETROLEUM" OR "AGIP%20PETROLEUM%20CO%20INC"…………………

 

  1. Total Fetched Records: 246

 

  1. Displaying only one Sample Record : {'id': 'osdu:master-data--Organisation:TGS', 'kind': 'osdu:wks:master-data--Organisation:1.0.0', 'version': 1, 'namespace': 'osdu:wks', 'authority': 'osdu', 'source': 'wks', 'type': 'master-data--Organisation', 'acl': {'owners': ['osdu.owners@tgs.com'], 'viewers': ['osdu.viewers@tgs.com']}, 'legal': {'otherRelevantDataCountries': ['US'], 'legaltags': ['odsu-tgs-dataset'], 'status': 'compliant'}, 'tags': {'DataLakeType': '', 'UpdatedCounter': '0'}, 'createTime': '2024-11-15T15:48:16+00:00', 'createUser': 'R360customerSupport@tgs.com', 'modifyTime': '2024-11-15T15:48:16+00:00', 'modifyUser': 'R360customerSupport@tgs.com', 'data': {'OrganisationName': 'TGS', 'Source': 'TGS'}}

 

  1. ERROR: {'status': 'error', 'message': Exception('Failed to create records from the workflow. Received status: 500. Received content: b\'{"code":500,"reason":"Unable to Enqueue workflow run","message":"Status code 413, \\\\"\\xef\\xbb\\xbf<?xml version=\\\\"1.0\\\\" encoding=\\\\"utf-8\\\\"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit._RequestId:373eedfa-c003-006f-2ee1-378ab6000000_Time:2024-11-16T04:41:02.5731809Z</Message><MaxLimit>131584</MaxLimit></Error>\\\\""}\'')}

Traceback (most recent call last):

File "/home/airflow/.local/lib/python3.8/site-packages/osdu_airflow/eds/eds_ingest/src_dags_fetch_and_ingest.py", line 80, in fetch_and_ingest

status = data_ingestor.load_data(

File "/home/airflow/.local/lib/python3.8/site-packages/osdu_airflow/eds/eds_ingest/data_ingestor/data_ingestor.py", line 11, in load_data

response = osdu_data_ingestor_obj.ingest_records(cleaned_records, criteria_dict)

File "/home/airflow/.local/lib/python3.8/site-packages/osdu_airflow/eds/eds_ingest/data_ingestor/implementation/osdu_data_ingestor.py", line 54, in ingest_records

raise Exception(

Exception: Failed to create records from the workflow. Received status: 500. Received content: b'{"code":500,"reason":"Unable to Enqueue workflow run","message":"Status code 413, \\"\xef\xbb\xbf<?xml version=\\"1.0\\" encoding=\\"utf-8\\"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit._RequestId:373eedfa-c003-006f-2ee1-378ab6000000_Time:2024-11-16T04:41:02.5731809Z</Message><MaxLimit>131584</MaxLimit></Error>\\""}'

CC : @AshishSaxenaAccenture , @ekay , @Atul, @priyankabhongade , @jeyakumar-jk