EDS bug in fetch_client.py
From airflow eds_ingest fetch_client on Azure. When attempting to pull from GCP M21 Preship.
Reading remote log from wasb-airflowlog/opendes:master-data--ConnectedSourceDataJob:AWSPreshipSH202311__2023-11-14T16:05:00+00:00/eds_ingest/fetch_client/2023-11-14T16:05:00+00:00/None/1.log.
While it's possible the credentials are no longer good or other issues. fetch_client.py is not checking return status and assuming a 200 return and "results" in the response (hence the KeyError
).
[2023-11-14, 16:05:08 UTC] {check_oauth_flow_type.py:28} INFO - Auth Code Generated for RefreshTokenKeyName
[2023-11-14, 16:05:08 UTC] {check_oauth_flow_type.py:29} INFO - _build_auth_code_request---end
[2023-11-14, 16:05:08 UTC] {fetch_client.py:61} INFO - QUERY BODY:{'kind': 'osdu:wks:master-data--Basin:1.0.0', 'query': '* AND ((createTime: [2023-02-01T00:00:00 TO 2023-11-14T16:05:06]) OR (modifyTime: [2023-02-01T00:00:00 TO 2023-11-14T16:05:06]))', 'sort': {'field': ['createTime'], 'order': ['ASC']}, 'limit': 5}
[2023-11-14, 16:05:08 UTC] {src_dags_fetch_and_ingest.py:152} ERROR - {'status': 'error', 'message': KeyError('results')}
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/osdu_airflow/eds/eds_ingest/src_dags_fetch_and_ingest.py", line 92, in fetch_and_ingest
records = self.fetch_client.fetch_details(criteria_dict,self.auth)
File "/home/airflow/.local/lib/python3.8/site-packages/osdu_airflow/eds/eds_ingest/implementation/fetch/fetch_client.py", line 67, in fetch_details
records = search_body["results"]
KeyError: 'results'