[M18] Authorization failure in EDS Ingest DAG
If authorization fails due to incorrect secret value, eds ingest fails by uncaught exception. Stack trace:
[2023-10-06, 13:00:27 UTC] {{src_dags_fetch_and_ingest.py:488}} ERROR - {'status': 'error', 'message': KeyError('value')}
Traceback (most recent call last):
File "/usr/local/airflow/dags/eds_ingest/libs/src_dags_fetch_and_ingest.py", line 465, in _build_auth_code_request
auth_code_authorization_code = self.generate_token.build_auth_code_authorization_code(self.auth)
File "/usr/local/airflow/dags/eds_ingest/libs/generate_tokens.py", line 67, in build_auth_code_authorization_code
value_data = self._get_secrets_values(secret_key)
File "/usr/local/airflow/dags/eds_ingest/libs/generate_tokens.py", line 26, in _get_secrets_values
secret_value = response_json['value']
KeyError: 'value'
Authorization can also cause the DAG to fail with other non-obvious errors:
File "/usr/local/eds/airflow/dags/eds_ingest/libs/src_dags_fetch_and_ingest.py", line 88, in fetch_and_ingest
connected_source_data_job = self.fetch_connected_source_registry_record.fetch_connected_source_data_job(
File "/usr/local/eds/airflow/dags/eds_ingest/libs/fetch_registry_entry_record.py", line 71, in fetch_connected_source_data_job
return search_response['results'][0]
KeyError: 'results'
Only after adding additional logging to the code the reason of the error can be found:
[2023-10-06, 16:58:00 UTC] {fetch_registry_entry_record.py:70} INFO - SERACH RESPONSE: {'reason': 'Access denied', 'status': 401, 'message': 'The user is not authorized to perform this action. Invalid token.'}