upgrade airflow, airflow charts and python version

Version changes

• Airflow version upgrade 2.4.3 --> 2.7.3
• Airflow community charts upgrade 8.5.2 --> 8.9.0
• Python version upgrade 3.8 --> 3.11 via osdu/platform/deployment-and-operations/base-containers-azure/airflow-docker-image!37 (merged)

Changes

• Upgraded python dependencies
• Added patch permission for pods in podOperator namespace (airflow)
• Disabled airflow triggerer deployment and using default triggerer.
• Related to osdu/platform&32 (closed)

Upgrade instructions

• Scaledown workflow service to 0 replicas , remove eds_scheduler cron ( scheduler_interval variable in airflow UI ) and makesure no dags are running during upgrade.
• Airflow DB migrations might take considerable amount of time, based on size of data. ( for GLAB and STG instances, it took around 9 minutes. )

Testing

• Changes are tested in community instances ( GLAB and dev secondary env. ) on EDS, CSV-parser and segy-to-zgy-conversion dags. No issues observed.

Performance Testing

• Tested 1000 runs of a dummy dag on 2.7.3 version
• Dag: parallel_dags.py
• Time taken for 1000 runs: 21 minutes, No failures obeserved, no tasks struck in waiting stage.
• Autoscaling is working as expected.