Continuous Batching
why your LLM server sits at 20% GPU and still feels slow
AI engineeringAI engineering / inference serving
The symptom
You self-host a 7B model behind vLLM/TGI on one A100. Load test: ~4 req/s, p99 latency 9s. nvidia-smi says 22% utilization. You raise batch size → throughput barely moves and latency gets worse. You add a second GPU → still bad.
The GPU isn't busy. So what is it waiting on?
The bug: static batching
Naive serving groups N requests into a batch and runs them lockstep until ALL are done. The batch finishes when the LONGEST generation finishes.
Batch of 8, output lengths:
[20, 25, 30, 40, 50, 60, 700, 800]The batch runs 800 decode steps. Slot-steps available: 8 × 800 = 6400. Slot-steps doing useful work: 1725.
→ ~73% of your GPU's decode capacity is spent computing padding for finished sequences.
And the 20-token request? It was done in 0.4s but returns after 16s, because it can't leave the batch.
The fix: schedule per iteration, not per batch
Decode emits ONE token per step, and each sequence owns its own KV cache. So batch membership doesn't have to be fixed — it can change every single step.
Continuous batching (a.k.a. iteration-level scheduling):
- sequence hits EOS → evicted immediately, slot freed that step
- queued request → joins on the very next step, no waiting for the batch to drain
Same workload, same GPU: 2–4× throughput, and short requests stop being held hostage by long ones.
The parts people get wrong
1️⃣ Your real limit is KV cache memory, not max_num_seqs
Concurrency stops where GPU KV blocks run out. Set --max-model-len 32768 when your traffic is 2k prompts and you've reserved room for giants you never serve. Trim it and concurrency jumps.
2️⃣ Prefill stalls decode
A new 8k-token prompt joining the batch triggers one heavy prefill step. Every streaming user's tokens hitch at that moment — smooth output, then a visible stutter. Fix: --enable-chunked-prefill splits it across steps.
3️⃣ Preemption is a silent throughput cliff
When KV memory fills, running sequences get preempted and later recomputed. Throughput drops while the dashboard still looks "healthy."
Watch these, not just RPS:
vllm:num_requests_running / num_requests_waiting
vllm:gpu_cache_usage_perc # >90% sustained = you're about to preempt
vllm:num_preemptions_total # should be ~0A reasonable start:
--max-num-seqs 256 --max-model-len 8192 \
--gpu-memory-utilization 0.92 --enable-chunked-prefillMental model
Static batching is a bus: it leaves when the last passenger arrives, and stops at every stop even if the seat is empty.
Continuous batching is a taxi rank: a seat opens, the next person takes it, that step.
Your exercise
Fire 100 concurrent requests with mixed lengths — 50 with max_tokens=16, 50 with max_tokens=512. Measure separately:
- TTFT (time to first token)
- inter-token latency
- end-to-end p99 for the SHORT requests only
Then rerun with --max-model-len cut in half. If short-request p99 falls sharply, you were queueing on KV blocks, not on compute. What would you tune first if TTFT got worse while throughput improved?