[Breaking Change] Zonal Redundancy for Airflow
Point Airflow to use the newly created Zone Redundant Redis Instance for future purposes. This will break the existing Airflow Runs and they will need to be triggered again.
Consuming this Change:
This change will break the existing Airflow Runs. If they can be retriggered without losing any data, just retrigger the Airflow Runs once this change is merged.
If retriggering end to end runs is not possible due to any reason and we don't want to lose the existing runs, there are 2 suggested methods:
-
Drain the entire Queue by do not sending any new requests to Airflow. Once the queue is drained, take the changes for pointing to the new queue(new redis instance) and resume the traffic to Airflow.
-
Stop sending any new Requests to Airflow and take the changes for pointing to the new Queue(new Redis instance). Now requeue all the tasks from the old queue into the new queue. Resume the traffic to Airlfow.
Prefer the first option to the second one as the second option has a big overhead of requeuing and may still result in data loss.