A developer at a mid-size startup injected a timestamp field into the top of their system prompt for debugging. Useful for logs. Catastrophic for their bill: every single API call was treated as a cold start, and their prompt caching hit rate dropped to zero. One field, one bad placement, $1,300 gone in a month.
This is the thing nobody tells you about prompt caching: it does not cache your prompt. It caches the computational state the model builds while reading it. And that state is fragile in specific, learnable ways.
What the model is actually doing during "prefill"
Before the model writes a single output token, it has to read your entire prompt. This is called the prefill phase. LLM inference consists of two distinct phases: the prefill phase, where the model processes the input prompt and generates attention key-value (KV) tensors, and the decode phase, where the model autoregressively generates output tokens. During prefill, the model computes attention over the entire input sequence, producing KV tensors that capture the contextual representations needed for subsequent generation.
Those KV tensors are the cache. The KV cache is the underlying state and inference mechanism. The prompt cache is the strategy or product capability that reuses these preprocessing results across requests. Most prompt cache implementations rely on reusing pre-computed K/V states.
Here is the part that changes how you think about it: when people hear "cache," many immediately think of an output cache, where a previously answered question is simply returned. But the prompt cache is not that kind of output cache. Even if there is a cache hit, the model will still regenerate the response. The prompt cache reuses intermediate results from the prefill phase for prompt prefixes.
Think of it like a book you have already highlighted. The model is not re-reading from page one - it picks up from the last stable margin note. If a subsequent prompt has a matching prefix with a cached prompt, the KV cache for the matching prefix can be retrieved from the cache. Cache hits will tend to have a faster time to first token (TTFT), and reusing the KV cache for matching prefixes exactly preserves model behavior even when prompt suffixes differ.
How OpenAI and Anthropic do it differently
Major LLM providers have implemented prompt caching with varying approaches. OpenAI offers automatic prompt caching on GPT-4o and newer models, where caching activates automatically for prompts exceeding a minimum token threshold, with cache hits occurring only for exact prefix matches. Anthropic provides developer-controlled caching through explicit cache breakpoints, allowing users to specify which portions of their prompt should be cached, with configurable time-to-live (TTL) options.
The economic gap between the two is material:
| OpenAI | Anthropic (Claude Sonnet) | |
|---|---|---|
| Activation | Automatic, no code changes | Manual cache_control breakpoints |
| Cached token discount | ~50% off input | ~90% off input ($0.30/M vs $3.00/M) |
| Write cost | None | 1.25× standard (5-min TTL) or 2× (1-hr TTL) |
| Minimum prefix | 1,024 tokens | 1,024 tokens (Sonnet/Opus) |
| Cache TTL | 5-10 min (up to 1 hr) | 5 min default or 1 hr explicit |
| Hit rate guarantee | ~50%, best-effort | 100% when prefix matches exactly |
Anthropic's design assumes you know your workload and want maximum savings on it. OpenAI's design assumes you'd rather not think about it.
For the break-even math: with the 5-minute TTL (1.25× write, 0.10× read), you save the write premium back after one cache hit. Hit once, you have already broken even. Every hit after that is pure savings.
One concrete production outcome: ProjectDiscovery raised their cache hit rate from 7% to 84%, cutting total LLM spend by 59-70% with a single architectural change. The architectural change was prompt ordering, not model switching.
To run the numbers yourself: a team running 2,000 Slack bot conversations per day, each carrying a 10,000-token system prompt, pays roughly $1,800/month on that prefix alone at standard Claude Sonnet pricing. At a 75% cache hit rate, the effective cost drops to around $495/month - a 72% reduction, from zero feature changes.
The rule that determines whether your cache hits
Cache matching is prefix-only and byte-exact. The main constraint is prefix matching. Prompt caching works by comparing the beginning of your current prompt against what is already cached.
This means prompt order is a first-class engineering decision, not a stylistic one. Keep the most stable content first: system prompt, then tool definitions, then conversation history, then the newest tool results. Volatile content always goes last.
The invisible killers:
Timestamps injected early. Customers accidentally invalidate their cache by including a timestamp early in their request for later lookup or debugging. Move that to metadata where it will not impact the cache.
Dynamic tool lists. MCP servers that connect mid-session, or tools that are loaded lazily, change the tool definitions - and with them, the prefix.
Non-deterministic serialization. Unordered JSON keys in tool schemas, floats formatted differently, a set iterated in hash order - if your prompt is not byte-stable across processes, your cache is not either.
Firing parallel requests before warming the cache. A cache entry only becomes available after the first response begins. If you fire 10 concurrent requests, only the first one writes the cache. The rest all miss. Send one request, wait for the first token, then fan out.
There is also a temporal failure mode specific to agents with human approval steps: Anthropic's default cache TTL is 5 minutes. If your agent takes longer than that between steps - waiting on human approval, running a slow tool, hitting rate limits - the cache expires. The 1-hour TTL costs 2× base input price to write, but prevents full recomputation on every delayed step.
Why this matters more as agents get bigger
The average prompt token count grew nearly 4× between early 2024 and late 2025, from roughly 1,500 tokens to 6,000 per request. Longer prompts make caching more valuable, not less.
Agent prompts grow even faster. A system prompt that starts at 2,000 tokens quickly balloons with tool definitions, retrieved documents, and conversation history. Agentic systems involve multi-step reasoning loops where the model calls tools, receives results, reasons about them, and calls more tools. In multi-agent setups where an orchestrator spawns sub-agents, prompt caching becomes especially powerful - sub-agents sharing the same base prompt can all benefit from a single cache write.
Beagle in action#product-support, 10:22amOne pattern worth noting: Anthropic's default TTL is 5 minutes, workspace-scoped. Prompt caching and Anthropic's Batch API stack - Batch gives 50% off all tokens, and combining it with cache reads can reach 95% savings on the repeated portion. That number is not a marketing projection; it reflects compounding two separate discounts.
Prompt caching: common questions
What does prompt caching actually cache?
It caches the key-value tensors the model computes during the prefill phase - not the text of the prompt. When a later request shares an identical prefix, the model skips recomputing that portion and loads the tensors directly. The response is still generated fresh; only the input processing is skipped.
Does prompt caching change the model's output?
No. Reusing the KV cache for matching prefixes exactly preserves model behavior even when the prompt suffixes differ. The output is deterministic given the same suffix; caching the prefix has no effect on generation.
When does prompt caching not help?
Three workloads where the caching investment does not pay back: short, mostly-variable requests. A 300-token classification prompt has nothing meaningful to cache. Also low-volume workloads where you never hit the same prefix twice, and any workflow where dynamic content appears before your static content in the prompt.
How is Anthropic's explicit caching different from OpenAI's automatic caching?
OpenAI prompt caching is fully automatic - zero code changes, 50% cost discount on cached tokens, approximately 50% hit rate (best effort, not guaranteed). Anthropic prompt caching is manual - you set cache_control breakpoints, get a 90% cost discount on cache reads, and a 100% guaranteed hit rate when configured correctly.
What is the most common mistake teams make with prompt caching?
Putting volatile data before stable data in the prompt. Do not put timestamps, request IDs, random ordering, or user-specific volatile data before stable policy and tools. One changing token near the top of a system prompt invalidates the entire cached prefix on every call.