Little's Law.

🕵️The scene

Friday. Checkout API. Traffic doubles from 250 → 500 req/s and p99 goes from 300ms to 9 seconds. CPU is at 30%. Memory is fine. The DB says it's bored.

Someone says "the service is slow, scale it up." Wrong lever. Here's the one line that tells you the real one.

📐The law

L = λ × W

L = things in the system (in-flight requests)

λ = arrival rate (req/s)

W = time each one spends inside (seconds)

That's it. No assumptions about distribution, burstiness, or scheduling. It holds for any stable system.

🔢Run the numbers on the scene

Before: λ=250/s, W=0.2s → L = 250 × 0.2 = 50 in-flight requests.

After: λ=500/s, W=0.2s → L = 100 in-flight requests.

Now look at the config nobody reread in a year:

DB_POOL_MAX = 60

You need 100 concurrent slots. You have 60. The other 40 requests aren't working — they're standing in line for a connection. That waiting is part of W, so W grows, so L grows, so the line gets longer. That's the 9 seconds.

CPU was at 30% the whole time because your threads were blocked, not busy. Idle CPU during a latency spike is the fingerprint of a concurrency limit, not a capacity limit.

🎯The flip: solve for what you actually control

Rearrange it. Given a pool of N and a service time W, the most throughput you can ever get is:

λ_max = N / W

60 connections ÷ 0.2s = 300 req/s. Hard ceiling. You were asking for 500. No amount of bigger instances moves that number — only more connections, or faster queries.

Same math sizes everything: worker threads, HTTP client pools, Kafka consumer concurrency, goroutines, Lambda reserved concurrency.

⚠️The part intermediates get wrong

"So I'll set DB_POOL_MAX = 400 and be safe."

No. Little's Law is descriptive, not a promise. It tells you how many are inside — not that the system can serve them. If Postgres does its best work at ~40 active queries, opening 400 connections just moves the queue from your app into the database, where you can't see it or shed it. Throughput stays flat, latency gets worse, and now you've lost the backpressure signal.

The rule: size the pool to what the downstream can actually absorb, then reject or shed above it. A fast 503 beats a 9-second success.

Two more traps:

• W is measured from arrival, not from when work starts. Queue time counts. If you only instrument handler duration, W looks fine while users suffer.
• The law needs a stable system (λ ≤ capacity). If arrivals exceed service rate, L isn't a number — it grows until something breaks.

🧪Your exercise

Pick one service you run. Find three numbers:

1. peak λ (req/s from your dashboard)
2. mean W (avg latency in seconds — not p99)
3. your configured concurrency limit (pool size / worker count)

Compute λ × W. Is it under your limit?

Then compute limit ÷ W — that's your true throughput ceiling. Is peak traffic anywhere near it?

Bonus: what's the smallest traffic increase that would push L past your limit? That number is how much headroom you actually have — and it's usually smaller than people guess.