Errors with running tests for Ingestion DAGs
Expected behavior:
Unit tests and end-to-end tests should work for core code out of the box.
Observed behavior:
When attempting to run unit and end-to-end tests for Ingestion DAGs core code, the tests in /plugin-unit-tests/ fail due to an issue related to Airflow's "variable" table ( see the error text below ).
From researching this issue, it appears that this could be related to needing to initialize the Airflow DB?
===================================================== test session starts ======================================================
platform darwin -- Python 3.8.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/spivbrad/Documents/aws-osdu-code/os-ingestion-dags
plugins: anyio-2.2.0, dash-1.19.0, mock-3.5.1
collected 79 items / 1 error / 78 selected
============================================================ ERRORS ============================================================
_____________________________ ERROR collecting tests/plugin-unit-tests/test_process_manifest_r2.py _____________________________
../../../.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1705: in _execute_context
self.dialect.do_execute(
../../../.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py:681: in do_execute
cursor.execute(statement, parameters)
E sqlite3.OperationalError: no such table: variable
The above exception was the direct cause of the following exception:
tests/plugin-unit-tests/test_process_manifest_r2.py:27: in <module>
from operators import process_manifest_r2
src/plugins/operators/process_manifest_r2.py:37: in <module>
config.read(Variable.get("core__config__dataload_config_path"))
../../../.local/lib/python3.8/site-packages/airflow/models/variable.py:123: in get
var_val = Variable.get_variable_from_secrets(key=key)
../../../.local/lib/python3.8/site-packages/airflow/models/variable.py:181: in get_variable_from_secrets
var_val = secrets_backend.get_variable(key=key)
../../../.local/lib/python3.8/site-packages/airflow/utils/session.py:65: in wrapper
return func(*args, session=session, **kwargs)
../../../.local/lib/python3.8/site-packages/airflow/secrets/metastore.py:66: in get_variable
var_value = session.query(Variable).filter(Variable.key == key).first()
../../../.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py:2695: in first
return self.limit(1)._iter().first()
../../../.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py:2779: in _iter
result = self.session.execute(
../../../.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py:1653: in execute
result = conn._execute_20(statement, params or {}, execution_options)
../../../.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1520: in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
../../../.local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py:313: in _execute_on_connection
return connection._execute_clauseelement(
../../../.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1389: in _execute_clauseelement
ret = self._execute_context(
../../../.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1748: in _execute_context
self._handle_dbapi_exception(
../../../.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1929: in _handle_dbapi_exception
util.raise_(
../../../.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py:198: in raise_
raise exception
../../../.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1705: in _execute_context
self.dialect.do_execute(
../../../.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py:681: in do_execute
cursor.execute(statement, parameters)
E sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: variable
E [SQL: SELECT variable.val AS variable_val, variable.id AS variable_id, variable."key" AS variable_key, variable.is_encrypted AS variable_is_encrypted
E FROM variable
E WHERE variable."key" = ?
E LIMIT ? OFFSET ?]
E [parameters: ('core__config__dataload_config_path', 1, 0)]
E (Background on this error at: http://sqlalche.me/e/14/e3q8)
=================================================== short test summary info ====================================================
ERROR tests/plugin-unit-tests/test_process_manifest_r2.py - sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no su...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================= 1 error in 1.18s =======================================================
A recommended solution:
The README for the tests mention a docker image that is used for testing. 1) could this image be made available for cloud providers to run these tests in a consistent environment? 2) could we alter the dockerfile used to run the unit_tests.sh
, set_airflow_env.sh
, and test_dags.sh
scripts?
Is this feasible?