Circuit Breaker Pattern.

❌The myth

"If a downstream service is flaky, add retries. More attempts = more resilient."

✅Actually

During a real outage, retries are gasoline. Service B is struggling at 2s latency. Service A retries 3x per call — so A now sends 3x the traffic to a service already on its knees. That's a retry storm. B falls over completely, A's threads pile up blocked on timeouts, and the failure cascades upward. This is how ONE slow dependency takes down an entire system.

A circuit breaker is the fix: stop calling a failing service fast, and give it room to recover.

🔌The mental model

Like an electrical breaker. It wraps calls to a dependency and has 3 states:

• CLOSED (normal) — calls pass through; count failures.
• OPEN (tripped) — failures crossed a threshold. Calls fail INSTANTLY without touching the network (return cached data / a default / a fast error). No thread sits blocked on a 30s timeout.
• HALF-OPEN (probing) — after a cooldown, let ONE test call through. Success → CLOSED. Fail → back to OPEN, reset the timer.

⚙️The logic in ~10 lines

if state == OPEN:
    if now - opened_at > cooldown:   # e.g. 30s
        state = HALF_OPEN
    else:
        raise CircuitOpen()          # fail fast, no call
try:
    r = call_downstream()
    if state == HALF_OPEN: state = CLOSED; failures = 0
    return r
except:
    failures += 1
    if failures >= threshold:        # e.g. 5
        state = OPEN; opened_at = now
    raise

📊Why it matters (numbers)

Without a breaker: 1000 req/s each blocking 30s on a dead service → thread pool exhausted in seconds → your service is down too.

With a breaker: after 5 failures it opens; the next 1000 req/s fail in <1ms, so your threads stay free to serve everything else. You degrade one feature instead of crashing the whole app.

🪤Gotchas practitioners get wrong

• Breaker + retry belong TOGETHER, not either/or: retry for blips, breaker for sustained failure. Only retry while CLOSED.
• Trip on error RATE (e.g. >50% of the last 20 calls), not a raw count — 5 errors out of 10,000 is noise.
• Always define the fallback. An open breaker still needs an answer: cached value, empty list, queued write. "Fail fast" must fail usefully.
• One breaker PER dependency, not one global — a slow analytics call shouldn't open the breaker for payments.

🎯Your exercise

Extend the 10-line sketch to trip on error rate: open only if ≥50% of the last 10 calls failed (hint: a small ring buffer of outcomes). Then answer this — why does HALF-OPEN allow only a single probe call instead of letting all traffic back at once?