Temperature 0 vs Deterministic.

🧨The myth

"Set temperature=0 and the LLM becomes deterministic — same prompt in, same output out, every single time." Almost everyone believes this. Then they write a test asserting exact string equality against a saved response, and it flakes in CI a week later.

✅Actually…

temperature=0 makes sampling GREEDY: at each step the model picks the single highest-probability token instead of sampling from the distribution. That removes the randomness from the sampler. It does NOT make the whole system deterministic.

🔍Why it still drifts

1. Floating-point math isn't associative. (a+b)+c ≠ a+(b+c) at the bit level. GPUs sum thousands of values in parallel, and the reduction order shifts with batch size, sequence length, and which kernel the scheduler picks. When two candidate tokens have almost-equal logits, a tiny rounding difference flips which one is "highest."
2. Batching / server load. Your request is batched with other users'. Batch size changes matrix shapes → different kernels → different rounding. So the SAME prompt at 3am (quiet) vs peak traffic can resolve differently.
3. MoE routing. Large mixture-of-experts models route tokens to experts in a way that can depend on the whole batch, not just your sequence.
4. "Same input" is a lie. A silent model-version bump behind a stable name, a tweaked system prompt, or a different default seed all break the premise.

🧪Quick proof

Fire the same temp=0 prompt 50 times at a hosted API:

for i in $(seq 50); do
  curl -s ... -d '{"temperature":0,"messages":[...]}' \
    | jq -r '.content'
done | sort | uniq -c

You'll typically see one dominant answer plus a few one-off variants — not 50 byte-identical strings.

🛠What to do instead

• Don't assert exact equality in tests. Assert STRUCTURE (valid JSON, required fields present), or use an embedding-similarity threshold, or an LLM-as-judge.
• Need reproducibility? Pin the model version explicitly, set a seed if the API exposes one (helps, still not guaranteed on shared infra), and snapshot outputs as approval files you re-bless when they legitimately drift.
• temp=0 is still the right call for extraction/classification — but for the OUTPUT QUALITY (less rambling, fewer wild tokens), not for byte-identical results.

⚖️Mental model

Temperature controls the SAMPLER, not the HARDWARE. You silenced one source of randomness; the numerical and serving stack underneath still has its own.

🎯Your exercise

You run a temp=0 endpoint that classifies support tickets into one of 5 labels, and a teammate wants to cache responses by hashing the prompt.

1. Is the cache safe? Before caching, why might two identical prompts have produced different labels?
2. Once you add the cache, where does determinism actually come from — the model, or something else?