Implement Integration tests for Ingestion DAGs
Create the testing stage in GitLab CI/CD, where the DAGs are tested inside a docker container.
The steps of the task:
- create container with Airflow-server and Flask server mocking external API (Storage and Workflow)
- Trigger the DAGs inside the container with different arguments
- Compare expected and actual results of DAG executions
- Integrate the steps mentioned above into GitLab CI/CD