(Hopefully) During the first half of Q2, We'll be performing a traffic failover to test that our procedures are ready and that we can failover in an emergency.

This task is to track the event. We will fail over to boston and back to london in one window lasting a few hours.

Procedure

• remove LB session affinity
• ensure Boston is pooled in web-global
• ensure web things are using web-global
• depool bots-London
• ensure MariaDB is Read-Only in both DCs
• switch the DB in phab
• depool tools-London
• promote tools-boston to tools primary
• test that the various web services work
• promote tools-London back to primary
• switch DB back to london in phab
• repool bots-london
• remove read only for DB & Phab
• raise session affinity back so traffic flows to london