Implement Integration tests for Ingestion DAGs
Create the testing stage in GitLab CI/CD, where the DAGs are tested inside a docker container.
The steps of the task:
-
create container with Airflow-server and Flask server mocking external API (Storage and Workflow)
-
Trigger the DAGs inside the container with different arguments
-
Compare expected and actual results of DAG executions
-
Integrate the steps mentioned above into GitLab CI/CD
-
AWS -
Azure -
IBM -
GCP
Edited by Dmitriy Rudko