Keep only DAG files in the dags folder.
It would be great if the dags
-folder contains only the following files: src/dags/eds_scheduler/eds_scheduler_dag.py
and src/dags/eds_ingest/src_dags_fetch_ingest_scheduler_dag.py
.
Other files with utilities and custom Airflow operators are supposed to be stored in either plugins
-folder or in a separate Python-package.
There are two reasons to do so:
- Airflow Scheduler has to parse not DAG files each time; it takes extra time
- Possible import issues
You might find this link useful: https://airflow.apache.org/docs/apache-airflow/2.2.5/plugins.html