Deploy on GCP Composer Airflow (GONRG-888)
Main changes:
-
Cloud Composer can't properly process file with code in the plugins folder if these files are big, as it is in our case. So the libs folder was moved into the dags folder.
-
Add an extra steps to make File part of Manifests follow R2 schema in process_energistics_op.py.
-
Fix
convert_to_r2_schemas
in process_energistics_op.py. -
Some minor bug fixes in process_energistics_op.py.
Edited by Yan Sushchynski (EPAM)