A ten-turn agent session with a 5,000-token system prompt reprocesses that same prompt ten separate times if nothing is cached. That is 50,000 tokens of input just for the static instructions alone - before a single novel token enters the picture. The fix exists, it is supported by every major provider, and most teams are not using it correctly.
This is a post about prompt caching: what it actually stores, why the savings compound on agentic workloads, and what kills the hit rate without warning.
What the model is actually doing when it reads your prompt
When an LLM processes your prompt, it generates key-value (KV) cache entries in its attention layers - mathematical representations of the relationships between tokens. Normally, the model recomputes this KV cache on every request. Prompt caching stores it so the model can skip that computation on subsequent requests that share the same prefix.
That is the whole mechanism. It is not a lookup table of previous answers. It is not semantic similarity matching. It is a provider-managed feature built into the LLM API, not something you build yourself. The provider hashes the front of your prompt, checks if it has seen those exact tokens in that exact order recently, and if so, loads the pre-computed KV state instead of running the expensive prefill step again.
On a cache miss - the first request - the full prefix is computed through every transformer layer, the KV states are stored in cache, and then the new tokens are processed. On a cache hit, stored KV states are loaded directly, skipping the expensive recomputation, and only the new tokens are processed fresh. The result: cache hits cost roughly 90% less and return the first token two to five times faster.
One thing that surprises people: because the model recomputes nothing it has already processed, output is byte-identical - there is no quality trade-off. The saving applies only to input tokens; output tokens are never discounted by any provider's caching scheme.
How Anthropic and OpenAI implement it differently
The providers converge on the same KV-storage idea but diverge sharply on how you activate it.
OpenAI prompt caching is fully automatic - zero code changes, a 50% cost discount on cached tokens, approximately 50% hit rate (best effort, not guaranteed). Anthropic prompt caching is manual - you set cache_control breakpoints, get a 90% cost discount on cache reads, and a guaranteed hit rate when configured correctly.
| OpenAI | Anthropic | |
|---|---|---|
| Activation | Automatic above 1,024 tokens | Manual cache_control breakpoints |
| Cache read discount | 50% off | 90% off |
| Write cost | None | +25% on first write (5-min TTL) |
| Default TTL | Not published | 5 minutes, resets on each hit |
| Extended TTL | 24 hours (GPT-4.1 series) | 1 hour (2× write cost) |
| Hit guarantee | Best-effort | Guaranteed when configured correctly |
Anthropic's writes cost 1.25× the normal input rate; reads cost 10% of the normal input rate - a 90% discount. Cache TTL is five minutes by default, with a one-hour option at a higher write rate.
With the five-minute TTL, you save the write premium back after one cache hit. Hit once, you have already broken even. Every hit after that is pure savings.
Minimum token thresholds vary by model: 1,024 tokens for Claude Opus and Sonnet models, 2,048 for older Haiku models, and 4,096 for Claude Haiku 4.5. Anthropic allows a maximum of four breakpoints per request.
If your traffic is high-frequency and your prompts are large and stable, Anthropic's 90% discount will dominate the comparison; if your traffic is sporadic or your prompts evolve frequently, OpenAI's automatic 50% with no write premium is the cleaner choice.
Where agentic workloads get hit hardest
A simple chatbot with a static system prompt is an easy win: cache the prefix, done. Agentic loops are where this becomes critical - and where teams bleed money before they understand why.
Agents that loop - reasoning, tool call, observation, repeat - reprocess the same tool definitions and system prompt on every iteration. A 10-iteration agent loop without caching is roughly 10× the input cost of the underlying prefix. With caching, it is roughly 1× the prefix cost plus 10× the small incremental tokens that change per iteration. This is where the savings compound most aggressively.
ProjectDiscovery documented exactly what this looks like at scale. Their security testing agent, Neo, runs 20-40 LLM steps per task. A single complex task with Opus 4.5 could consume 60 million tokens. After implementing caching, overall caching saved 59% on LLM costs compared to what the same token volume would have cost at full input rates. Post-optimization that number is 66%, and the last 10 days peaked at 70%.
The surprising part is what actually produced those numbers. It was not tuning TTL settings or adding more breakpoints. One relocation raised the cache hit rate to 84% overnight and cut overall LLM cost by 59% - peaking at 70% in the final 10 days of the engagement, with 9.8 billion tokens ultimately served from cache. The lesson generalizes: the highest-value caching work is almost never tuning TTLs or breakpoints; it is auditing what dynamic content has crept into the supposedly static prefix.
That is the non-obvious failure mode. Dynamic content - a session ID, a timestamp, a rotating working-memory scratchpad - sitting in the system prompt makes the prefix different on every request. The cache never warms. On Anthropic, you still pay the 25% write premium on every request with no reads to offset it. You end up paying more than without caching.
Beagle in action#ai-infra, Tuesday, 11:02amrequest_id field that was moved into the system prompt three weeks ago during a debug session, drafts a message with the specific line and the projected monthly cost differenceThe four structural rules that keep cache hit rate high
Caches hit on prefixes. The placement principle: put everything stable first, everything variable last. The order that produces the best hit rate: system prompt (most stable), tool definitions (stable per agent version), long static context such as corpus excerpts that recur across queries.
After that, the user message goes last - it is the only part that should change between requests.
Stable before dynamic. If anything in your prefix changes per request, the cache cannot match from that point forward. One dynamic field high in the system prompt zeros the savings.
Watch the TTL. Anthropic's cache expires after roughly five minutes of inactivity. Each cache hit resets the timer. An active session - messages every minute or two - keeps the cache warm indefinitely. Stop for five minutes and the cache evaporates. For slower agent loops with human-approval steps, use the one-hour TTL.
Check the minimum threshold. The minimum is 1,024 tokens for OpenAI. The cache activates in 128-token increments above that floor. A 900-token system prompt will not cache.
Avoid summarizing tool call history. Techniques such as summarizing or pruning old tool calls break cached representations. The emerging pattern is to maintain a stable system prompt that benefits from caching while treating tool calls as dynamic content that may be managed throughout the session.
An AI teammate like Beagle, which runs the same system prompt across many in-Slack tasks, benefits directly from this structure - the static instructions and tool schemas stay cached across an entire workday if requests keep coming in within the TTL window.
Prompt caching LLM: common questions
What exactly does prompt caching store?
Prompt caching stores the computational state from an LLM's attention layers so the model can skip redundant prefill work on repeated prompt prefixes. The result is lower time-to-first-token and cheaper input costs on every request that hits the cache for a shared prefix. It stores intermediate math, not the text itself or any generated output.
Does prompt caching change the model's output?
No. It caches the internal state of the model for that prefix, reducing redundant computation. This results in reducing latency and input token savings, without any loss in quality. The model generates a fresh response every time; only the prefill work is skipped.
When does prompt caching not help?
Highly dynamic prompts where the entire prompt changes between requests drop the cache hit rate to zero. On Anthropic, you still pay the 25% write premium on every request with no reads to offset it - you end up paying more than without caching. Very short prompts (under the 1,024-token minimum) and one-shot interactions with no follow-up also see no benefit.
How is prompt caching different from semantic caching?
They operate at different layers. Prompt caching at the provider side reduces the cost of repeated static prefixes within a session. Semantic caching works at the client side, keyed on rendered prompts, and can eliminate redundant API calls entirely for identical queries across sessions - the two mechanisms are complementary.
Which provider gives better savings - OpenAI or Anthropic?
It depends on traffic patterns. If your traffic is high-frequency and your prompts are large and stable, Anthropic's 90% discount will dominate the comparison; if your traffic is sporadic or your prompts evolve frequently, OpenAI's automatic 50% with no write premium is the cleaner choice. At 1 million cached input tokens per day, the per-day difference between the two is roughly $4.50 in Anthropic's favor - small at low volume, meaningful at sustained scale.