DNS TTL & Failover Tail.

🐛The symptom

14:00 — primary region dies. You flip the A record to the standby IP. TTL on the record: 60.

14:01 — you resolve it yourself, get the new IP.

14:06 — dashboards still show ~30% of requests hitting the dead IP.

14:45 — still ~8%.

$ dig +noall +answer api.example.com

api.example.com. 60 IN A 203.0.113.9 ← new, correct

The record is right. The traffic is wrong. Let's find who's lying.

🔍Suspect 1 — the TTL you lowered today

You ran at TTL=3600 until 13:58, then dropped it to 60 as part of the incident response.

Too late. A TTL travels WITH the copy already handed out. Every resolver that fetched the record at 13:30 is holding it for up to an hour, and your new 60 never reaches them.

Rule: lower the TTL one full OLD-TTL period BEFORE the change. Lowering it during the change buys nothing.

🔍Suspect 2 — TTL is advice, not law

Nothing forces a resolver to expire on schedule, and plenty add their own layer:

• ISP / corporate resolvers often clamp a floor (30–300s) and ignore your 60
• Chrome keeps its own in-process cache
• systemd-resolved / nscd cache below that
• the JVM is the classic trap: java.security networkaddress.cache.ttl can be -1 = cache forever. A Java client that resolved you at boot will never look again. Set -Dnetworkaddress.cache.ttl=30.

Go and Node re-resolve per dial — but see suspect 3.

🔍Suspect 3 — the real one: open connections never re-resolve

DNS is consulted when a socket is OPENED. Never again.

Any client with an HTTP keep-alive pool that connected at 13:30 keeps writing to 203.0.113.4 for as long as the socket lives. TTL=0 changes nothing — there is no lookup to change.

The split that tells you which suspect you have:

• Hard crash → sockets break, clients re-resolve → your tail is suspects 1 & 2 (stale caches).
• Graceful cutover, old box still answering → suspect 3. Traffic keeps flowing to a "drained" host for hours, and DNS metrics look perfect.

✅The fix

1. Don't use DNS as your failover mechanism. Point DNS at a stable health-checked LB / anycast VIP and fail over BELOW DNS — seconds, not TTLs.
2. If you must move a record: pre-lower TTL a full old-TTL ahead, then budget 2–3× TTL for the tail, and keep the old IP alive proxying to the new one until it drains.
3. Cap connection LIFETIME, not just idle time — nginx keepalive_time 1h, Envoy max_connection_duration. An idle timeout never fires on a busy pool.

🧪Your exercise

Run this twice, five seconds apart, against your router/ISP resolver:

dig +noall +answer github.com ; sleep 5 ; dig +noall +answer github.com

Watch the TTL count DOWN — that's the remaining life of a cached copy, not the record's configured TTL. Now compare with dig @1.1.1.1 github.com (fresh).

Then the real question: pick one service you own and answer — if its IP vanished right now, what is the longest a real client could keep using the old one? Count every layer: authoritative TTL, recursive resolver, OS cache, runtime cache, and any keep-alive pool. That number, not your TTL, is your actual failover time.