[BUG] Airflow2 High Request Spec Causes Resource Limitation Issue during osdu-azure Deployment
Airflow2 installation has very high requests specs to accommodate airflow stress tests needs. However, these requests requirements specs could be too high (6 worker nodes x 2.5 CPUs per node), thus will consume most of the CPU for the AKS node internal limit (10 nodes X 4 CPUs per node), and will not allow creation of new resources when deploying osdu-azure, unless autoscale node limit is increased. This issue can be solved either by lowering the airflow2 specs, or implement by default autoscale on airflow2 instead of using those higher specs (recommended). Not mandatory, but recommended, the size of the VM nodes (current 4 CPUs X 32G) can be increased to 8 CPUs X 64G, and default number of nodes can be decreased from 5 to 3, with an up-limit of 10 nodes.