Deploy on GCP Composer Airflow (GONRG-888)
Main changes:
- Cloud Composer can't properly process file with code in the
pluginsfolder if these files are big, as it is in our case. So the libs folder was moved into thedagsfolder. If we don't want Airflow to search DAGs in the libs folder, we must add the.airflowignoreinto energistics/dags with specifying files we want to ignore. - Add an extra steps to make File part of Manifests follow R2 schema in process_energistics_op.py.
- Some minor bug fixes in process_energistics_op.py.
Edited by Yan Sushchynski [EPAM/GC]