Feature change - Helm values override support - Support of overriding helm config in airflow deployment

Why is this change needed

Customers requested to provide a way to override default helm-config.yaml that is checkedin into the repository. Airflow is deployed in AKS cluster using helm charts. For airflow helm charts configuration like database to connect to, redis, replicas etc., are provided using helm-config.yaml. This helm config is checkedin into master repository which is basic and tuned for gitlab and dev environment. Using the override feature customer can provide a override configuration file which will override default configuration provided in the repository. This is optional and customer can deploy the existing templates without any change.

Current Behavior

Currently customers are changing the helm-config.yaml file directly once forked from infrastructure repository. This is causing conflicts when the helm-config.yaml file is updated in the infrastructure repository and manual intervention is needed to resolve this.

Expected Behavior

Provide a way for customers to provide a override file without worrying about handling merge conflicts.

Design proposal

ADO pipelines are modified to take extra values files "helm-config-override.yaml" which is always empty in the infrastructure repository. ADO pipelines are modified to take two input value files "helm-config.yaml" and "helm-config-override.yaml". After this change customers who want to deploy airflow can give overriding values through "helm-config-override.yaml". As this file is always empty there won't be any conflicts.

Acceptance Criteria

• Design Feature to ensure can be implemented with a non breaking change.
• Update all required documentation