Graceful Shutdown (SIGTERM handling).

🎬The scenario

You ship a new version. Rolling deploy, zero errors in the dashboard... except your on-call gets 12 reports of "payment succeeded but order never created." Every deploy, a handful of in-flight requests just vanish. Retries "fix" it, so nobody digs in. This is the classic ungraceful shutdown bug.

🔍What's actually happening

When Kubernetes (or Docker, or a load balancer) stops a container, it does NOT kill it instantly. It sends SIGTERM — a polite "please wrap up." Then it waits (default 30s in k8s: terminationGracePeriodSeconds) and only then sends SIGKILL (unignorable).

Most apps never install a SIGTERM handler. So the default kicks in: the process dies immediately, dropping every request currently mid-flight — including that half-finished payment.

🧩The two-part fix

1. Catch SIGTERM and stop accepting NEW work, but let in-flight work finish.

Node/Express:

const server = app.listen(3000)
process.on('SIGTERM', () => {
  server.close(() => {   // stops new conns, drains active ones
    db.pool.end()
    process.exit(0)
  })
  setTimeout(() => process.exit(1), 25000) // hard cap < grace period
})

2. Handle the readiness race. Here's the subtle part most people miss:

When SIGTERM arrives, the pod is being removed from the Service endpoints — but that removal propagates asynchronously. For a brief window, the load balancer STILL routes new traffic to your "shutting down" pod. If you already called server.close(), those new requests get connection-refused.

The pro move: on SIGTERM, first flip your readiness probe to failing and sleep a few seconds so the LB stops sending traffic — THEN drain.

let shuttingDown = false
app.get('/ready', (_, res) =>
  res.sendStatus(shuttingDown ? 503 : 200))
process.on('SIGTERM', async () => {
  shuttingDown = true
  await sleep(5000)      // let endpoints update
  server.close(...)
})

⚠️Common mistakes

• Using SIGKILL-style exits (process.exit() with no drain).
• Grace period shorter than your longest request → SIGKILL cuts it anyway.
• Forgetting background workers: a queue consumer must stop pulling new jobs but finish the current one (ack only after completion).
• PID 1 problem: in a container, your app is often PID 1 and Linux does NOT apply default signal handlers to PID 1 — if you don't explicitly handle SIGTERM, it's ignored entirely and you always wait for SIGKILL. (Fix: an init like tini, or handle it yourself.)

🧠Mental model

SIGTERM = "last call at the bar." Stop serving new drinks, let current customers finish, then lock up. SIGKILL = the fire alarm — everyone out NOW, no cleanup.

🏋️Your exercise

Take any HTTP service you have. Add a route that sleeps 10s then responds. Start it, fire that request, and in another terminal send kill -TERM <pid>. Does the in-flight request complete, or die? Now add a SIGTERM handler with server.close() and try again. Then answer: what happens if your grace period is 8s but the request needs 10s — and how would you detect that in production?