Implement Integration tests for Ingestion DAGs

Create the testing stage in GitLab CI/CD, where the DAGs are tested inside a docker container.

The steps of the task:

  • create container with Airflow-server and Flask server mocking external API (Storage and Workflow)

  • Trigger the DAGs inside the container with different arguments

  • Compare expected and actual results of DAG executions

  • Integrate the steps mentioned above into GitLab CI/CD  

  • AWS

  • Azure

  • IBM

  • GCP

Edited by Dmitriy Rudko