Every time your app sends a 10,000-token system prompt to a model API, the model does the same expensive matrix arithmetic it did on the previous request. Pay for it once, fine. Send a 2,000-token system prompt, and the model reprocesses every token. Send the next turn, and it reprocesses them again. Twenty turns in, you have paid to compute the same static instructions twenty times over. Prompt caching is the mechanism that stops this.
It is also the highest-ROI optimization most teams skip, because it sounds like a caching layer you have to build. You do not. You mostly just need to understand what the providers are actually doing - and what quietly breaks it when you start running agents.
What the model actually stores between requests
Prompt caching stores the computational state from an LLM's attention layers so the model can skip redundant prefill work on repeated prompt prefixes. The result: lower time-to-first-token (TTFT) and cheaper input costs on every request that hits the cache for a shared prefix.
To understand what "computational state" means here, you need to know one thing about how transformer models process text. When a model reads your prompt, every token computes two matrices - a key (K) and a value (V) - that encode what it has seen so far. These K-V pairs are what the model consults as it generates each new token. Providers save these K and V matrices in their datacenters, and if you send a new request that starts with the same prompt, they reuse the cached K and V rather than recalculating them.
In decoder-only transformers, where each token attends only to previous tokens, reusing the KV cache for matching prefixes exactly preserves model behavior, even when the prompt suffixes differ. This is the non-obvious part: the output is mathematically identical to what a full recompute would have produced. You are not getting an approximation. You are skipping arithmetic the GPU has already done.
The rule is strict: the cached portion must be an exact byte-for-byte prefix of the new request. If you change anything in the cached region - even a single character - you get a cache miss.
How the three major providers implement this differently
Major LLM providers have implemented prompt caching with varying approaches. OpenAI offers automatic prompt caching on GPT-4o and newer models, where caching activates automatically for prompts exceeding a minimum token threshold, with cache hits occurring only for exact prefix matches. Anthropic provides developer-controlled caching through explicit cache breakpoints, allowing users to specify which portions of their prompt should be cached, with configurable time-to-live (TTL) options. Google offers both implicit caching, which activates automatically with no guaranteed cost savings, and explicit context caching, where developers create and reference caches with guaranteed discounts.
Here is what that looks like in practice:
| Provider | Opt-in required | Min threshold | Cache read discount | TTL |
|---|---|---|---|---|
| OpenAI | No (automatic) | 1,024 tokens | ~50% | 5-10 min (up to 1 hr) |
| Anthropic | Yes (cache_control) |
1,024 tokens | 90% | 5 min or 1 hr |
| Google Gemini 2.5 | No (implicit) or yes (explicit) | 1,024-2,048 tokens | 75-90% | Configurable |
| DeepSeek | No (automatic) | 64-token chunks | 90% | Automatic |
Sources: OpenAI, Anthropic, Google Cloud, DeepSeek
The cost difference between providers is real. Anthropic's prompt caching delivers a 90% discount on cache reads, reducing Claude Sonnet 4.6 input cost from $3.00/M tokens to $0.30/M tokens. OpenAI's automatic version gets you there with zero code changes - compared to Anthropic and Google, OpenAI's prompt caching is the most seamless, requiring no code changes.
To see what the savings actually look like at volume: a 10,000-token system prompt sent to Claude Sonnet at 2,000 requests per day with a 75% cache hit rate saves roughly $1,215 per month on input tokens alone. The breakeven math matters too - implementation details such as minimum token thresholds (typically 1,024-4,096 tokens depending on model), TTL durations (ranging from 5 minutes to 24 hours), and pricing structures vary across providers and are subject to change.
Why agents break the cache in ways chatbots do not
A single-turn chatbot has a simple caching story: keep the system prompt stable, everything else varies at the end. Cache warm, every time. Agents are harder.
Modern KV cache management assumes the chatbot workload: prompts arrive once and the cache grows append-only, so prefix caching and forward-only eviction are correct by construction. Agentic LLMs break this assumption. Their conversations evolve through policy-driven editing: failed tool calls are retried, stale outputs dropped, trajectories pivoted.
The specific failure mode worth knowing: identical content moves to new positions between turns, invalidating exact-prefix caches even though the underlying KV would still be valid. If your agent inserts a timestamp, a retrieved document, or a dynamic tool result anywhere before the end of the prompt on every turn, the cache misses. The model recomputes everything from that insertion point forward.
A 2026 evaluation of prompt caching on long-horizon agentic tasks (arXiv:2601.06007) put numbers on this. Prompt caching reduces API costs by 41-80% and improves time to first token by 13-31% across providers. Strategic prompt cache block control - placing dynamic content at the end of the system prompt, avoiding dynamic traditional function calling, and excluding dynamic tool results - provides more consistent benefits than naive full-context caching, which can paradoxically increase latency.
On Anthropic specifically, the discipline is stricter than "put dynamic data last." Tool definitions must remain byte-identical and in the same order across requests; changing tool_choice, thinking parameters, or an image in the system prompt invalidates downstream cache entries.
There is a second, subtler problem for agentic workloads: the cache evicts between tool calls. Agentic workloads characteristically interleave inference steps to derive the next action and execution steps where the agent calls an external tool. The output of the tool is subsequently appended to the request context, and a new inference step is initiated. The core issue arises after the request's KV cache is evicted when the agent transitions from inference step to tool call. If the tool call takes longer than the TTL - five minutes on most providers - the cache is gone and the agent pays full prefill cost on the next turn.
Beagle in action#engineering, automated daily standup agentHow to structure your prompts so the cache actually hits
The ordering rule is the whole game. Think of your prompt in layers from most-stable to least-stable, and put them in that order:
- System instructions - never change; goes first
- Tool definitions - change rarely; lock them in byte-identical order
- Retrieved documents or knowledge - changes per session, not per turn; cache with a longer TTL
- Conversation history - grows each turn; append-only keeps the prefix valid
- User's current message - always last; never cached, always fresh
Anthropic supports up to four cache breakpoints per request with a lookback window of up to 20 content blocks per breakpoint - enough to cache tools, system prompt, and a document corpus as separate stable segments while leaving the query uncached.
ProjectDiscovery raised their cache hit rate from 7% to 84%, cutting total LLM spend by 59-70% with a single architectural change. That is a documented production outcome, not a benchmark projection. The architectural change was prompt ordering - moving the large static system prompt ahead of dynamic content and keeping tool definitions stable.
One number worth computing before you start: Anthropic charges a small write premium when content is first cached. The break-even is two cache hits per cached block. At any meaningful request volume with a stable system prompt, you clear that threshold in the first minute of traffic.
Prompt caching LLM: common questions
What is prompt caching in LLMs?
Prompt caching stores the key-value attention matrices a model computes during prefill, then reuses them on subsequent requests that share the same prefix. The model skips recomputing those tokens entirely. Cache hits cost 50-90% less than fresh tokens and return the first token measurably faster - typically 13-85% lower TTFT depending on prompt length and provider.
Does prompt caching change the model's output?
No. In decoder-only transformers, reusing the KV cache for matching prefixes exactly preserves model behavior, even when the prompt suffixes differ. The cached computation is mathematically identical to a fresh run. You are not trading quality for cost - you are skipping redundant arithmetic.
How long does a cached prompt last?
TTL durations range from 5 minutes to 24 hours depending on provider and plan. OpenAI's default is 5-10 minutes of inactivity. Anthropic offers 5-minute or 1-hour TTLs, chosen per cache breakpoint. Google's explicit context caches are configurable. For agents with slow tool calls, the default 5-minute TTL can expire mid-session.
Why does my agent have a low cache hit rate?
The most common cause is dynamic content appearing early in the prompt. A cache hit occurs when the entire prefix matches exactly, allowing the system to reuse previously computed KV tensors. A cache miss occurs when any token differs from the cached content, even at the very beginning, forcing complete recomputation of all subsequent tokens. Timestamps, rotating document IDs, or tool output inserted before the static system content will kill the cache on every turn.
Is prompt caching the same as semantic caching?
No - they operate at completely different layers. Semantic caching returns a stored response for any query semantically similar to a previously answered one, skipping the model entirely. Prompt caching, by contrast, only skips the prefill computation - the model still runs and generates a fresh response. Semantic caching has a higher ceiling when it hits, but introduces accuracy risk. Prompt caching is lossless and works on every request that shares a stable prefix.