Skip to content
Shakeride

Guide · Updated

DNS failover that never fails over: what to check first

DNS failover depends on a chain: something decides the primary is unhealthy, the record set reacts to that decision, and clients ask again. A break anywhere in the chain looks the same from outside: traffic stays put.

How failover routing decides

With failover routing in Amazon Route 53, a primary and a secondary record share a name. Route 53 answers with the primary while it is considered healthy and with the secondary when it is not. Everything turns on how health is determined.

Checks, in order

Start where the decision is made, then move towards the client.

  • Is a health check associated with the primary record, or is health evaluated from an alias target? If neither, the primary is always treated as healthy.
  • Does the health check test what customers depend on, or a path that stays up when the application is down?
  • Are the record types, names and failover roles what you think they are? Look-alike records in the zone are easy to edit by mistake.
  • How long is the TTL? Clients keep a cached answer until it expires, so a long TTL delays every failover.

Test it before you need it

A failover setup that has never been exercised is a guess. Make the primary fail its health check on purpose, in a window you choose, and time how long clients take to move.

All guides

Practise on a real failure

A live AWS account, already broken, and a clock. Fix it with the console and command line, then read a report of how you worked.

Start your first ride

Browse the challenge catalog