Idle Timeout Mismatch (LB Keep-Alive vs Backend).

🧠The mental model

Keep-alive is a phone line left open between calls. Your load balancer keeps a pool of open lines to each backend so it doesn't pay a TCP+TLS handshake per request.

But both ends run their own egg timer for "nobody has spoken in a while, I'm hanging up." If the BACKEND's timer is shorter than the LB's, the backend hangs up a line the LB still believes is open. The next request goes down a dead line → 502. Nobody logged an error, because your app never saw that request.

📉The fingerprint

• 0.01%–0.5% of requests fail as 502, no pattern by route
• WORSE at low traffic (idle lines actually get to go idle)
• Backend access log has no entry at all for the failed request
• On AWS: HTTPCode_ELB_502_Count climbs while HTTPCode_Target_5XX stays flat

⚙️The rule, write it on the wall

backend keep-alive timeout > LB idle timeout

Defaults that collide straight out of the box:

• ALB idle timeout: 60s
• Node.js server.keepAliveTimeout: 5s

Node hangs up 55 seconds before the ALB expects it to.

Fix in Node:

server.keepAliveTimeout = 65000;  // > 60s ALB
server.headersTimeout    = 66000;  // must exceed keepAliveTimeout

nginx upstream: keepalive_timeout 75s (already >60 — which is why nginx rarely shows this bug).

Gunicorn: --keep-alive 65. Envoy / k8s ingress: same inequality, different field name.

🪓Where the analogy breaks — the intermediate part

1. Hanging up isn't instant. The FIN has to travel. The LB can write a request in the same microsecond the backend sends FIN. Matching the timers shrinks that race to near-zero; it does not delete it. So you still want retry on idempotent requests — and a POST you can't safely retry needs an idempotency key, not a timeout tweak.
2. Some middleboxes never hang up — they just stop forwarding. A NAT Gateway drops an idle flow after 350s with NO FIN. Your pooled DB connection looks alive forever, and the next query hangs until the OS TCP retransmit budget runs out (minutes, not ms). The fix here is the opposite tool: TCP keepalive BELOW their timer — Postgres keepalives_idle=60, or net.ipv4.tcp_keepalive_time=300 — so packets keep the flow warm.

So two different reflexes:

• Polite peers that send FIN → make the INNER timer longer.
• Silent packet-droppers → send traffic MORE OFTEN than their timer.

🔧Your exercise

Find the two numbers in your own stack: your LB/ingress idle timeout, and your HTTP server's keep-alive timeout. Which one is smaller? If it's the server's, you have a 502 generator running in prod right now.

Bonus: what's the idle timeout on your OUTBOUND path (NAT gateway, firewall) versus your DB pool's idle connection lifetime?