Prompt Caching — The Prefix Match You Keep Breaking.

🪲The symptom

You cached your 20K-token system prompt. The bill didn't move. Your logs:

req 1: cache_creation=20480  cache_read=0
req 2: cache_creation=20480  cache_read=0
req 3: cache_creation=20480  cache_read=0

No error. Every call succeeds. And you are now paying MORE than before caching: a cache write costs ~1.25x base input price, a read ~0.1x. You bought the write premium on every request and never once read it back.

🔍Step 1 — read the usage fields correctly

Total prompt = input_tokens + cache_creation_input_tokens + cache_read_input_tokens.

input_tokens is only the uncached remainder, not the whole prompt. cache_read pinned at 0 across requests that look identical means the prefix is not byte-identical.

🔍Step 2 — find the invalidator

Caching is a prefix match over rendered bytes. Render order is:

tools → system → messages

One byte differs at position N and everything from N onward is cold, no matter where you put your markers.

Usual suspects, ranked by how often they're the culprit:

• f"Today is {datetime.now()}" in the system prompt → brand-new prefix every request
• json.dumps(tool_schema) without sort_keys → key order drifts between runs
• tools=build_tools(user) → tools render at position 0, so nothing caches across users
• a session or user ID interpolated into system text → per-user prefix, zero sharing

Ours was line 1 of the system prompt: a timestamp. 20K tokens of perfectly stable instructions sat behind it, all uncacheable.

🔧 Step 3 — the fix: volatile content goes AFTER the last breakpoint

system=[{"type": "text", "text": STATIC_PROMPT,
         "cache_control": {"type": "ephemeral"}}],
messages=[*history,
          {"role": "user", "content": f"[now: {ts}] {question}"}]

A change at turn 5 invalidates nothing before turn 5. From req 2 on: cache_read=20480.

⚠️Two gotchas that bite AFTER it works

• Minimums are silent. Under the model's minimum cacheable prefix (512–4096 tokens, varies by model) you get cache_creation=0 and no error at all.
• Breakpoint on the wrong end. If your prompt is [shared 10K context][unique question] and you mark the very end, every request writes its own entry and reads nothing — a pure surcharge. Mark the end of the SHARED part.

💰The break-even math

5-min TTL: 1.25x write + 0.1x read = 1.35x for two calls, vs 2x uncached. Two requests and you're ahead.

1-hour TTL: write costs 2x, so you need three.

A read refreshes the entry's timer for free — so continuous traffic keeps a 5-minute entry warm indefinitely, and the 1h TTL buys you nothing there except a doubled write price. It only pays for 5-60 minute gaps.

🏋️Your exercise

Take any multi-turn LLM app you have and log usage on three consecutive turns. Check for the healthy signature: cache_read should GROW every turn while cache_creation stays small — roughly just what the last turn appended. If cache_creation is near the full conversation size on every request, something upstream is rewriting your history. Diff two consecutive request bodies and find the first divergence inside the overlapping region.