ProjectDiscovery's security agent Neo ran for months with prompt caching switched on and a real cache hit rate of 7%. According to their own writeup, caching showed as enabled on the dashboard the entire time. The bug wasn't exotic: mutable working memory sat inside the prefix that was supposed to be static, so every step invalidated the entire cached block behind it. One architecture fix later, the cache hit rate jumped to 84%, cutting overall LLM cost by 59%, with 9.8 billion tokens ultimately served from cache.
That gap-between "caching is on" and "caching is working"-comes down to a mechanism most developers never look at closely. This post explains it from the inside.
What the model is actually doing when it processes your prompt
Every time you send a request to a language model, the model has to turn your tokens into something it can reason over. During the prefill phase, the model processes the entire prompt. Each token attends to previous tokens via causal self-attention, calculating Query, Key, and Value tensors across all transformer layers to produce the first output token.
The Key and Value tensors are the expensive part. While producing one token at a time, the model must compute attention between the current token's query and the keys of all previous tokens. Naively, this would require recomputing the key and value projections for every past token at each generation step-a very expensive operation. The KV cache solves this by storing the key and value projections from previous tokens.
By reusing the stored K and V matrices, the model reduces inference time complexity from O(n²) to O(n), where n is the sequence length, since attention is computed only for newly generated tokens.
That's what happens within a single request. Prompt caching extends the same idea across requests. Think of the KV cache as a lookup table that lives in GPU memory-or increasingly, in server-side storage for API providers. If your current request starts with the same tokens as a previous one, the provider can skip recomputing those tensors entirely and serve them from the stored cache. For a 50,000-token context window where 40,000 tokens are cached, you're only doing roughly 20% of the attention compute.
That compute reduction is why providers can charge less. That discounted cached input rate isn't charity-it reflects the actual compute savings. When the provider doesn't need to run the full attention computation for your system prompt, they pass some of that saving along.
How Anthropic and OpenAI implement it differently
The discount is comparable; the mechanics are not. Some providers like OpenAI and DeepSeek offer automatic caching, while others including Google and Anthropic require manual setup.
With Anthropic, you are only charged for cache writes-when new content is written to the cache at 25% more than base input tokens for a 5-minute TTL-and cache reads, when cached content is used at 10% of base input token price.
If five minutes is too short, Anthropic also offers a 1-hour cache duration at additional cost.
A cache hit costs 10% of the standard input price, which means caching pays off after one cache read for the 5-minute duration, or after two cache reads for the 1-hour duration.
Anthropic lets you place up to four cache_control markers anywhere in the request-meaning a retrieved document inserted mid-conversation can still cache, and the system prompt plus tools can cache as a separate block from retrieved documents below it.
For RAG workloads, this is a real structural advantage. For straight chat with no document retrieval, OpenAI's automatic model is simpler and equivalently priced.
OpenAI does caching automatically, attempting to route requests to cached entries when possible. In one experiment, by sending a request and then immediately resending it, a hit rate of about 50% was achieved.
Anthropic gives you more control, letting you decide when to cache and for how long-and in testing, Anthropic routes to cached entries 100% of the time when you explicitly ask them to cache a prompt.
| Provider | Setup | Min tokens | Cache read discount | TTL options | Write surcharge |
|---|---|---|---|---|---|
| Anthropic | Manual cache_control |
1,024 | 90% off | 5 min, 1 hour | +25% on writes |
| OpenAI | Automatic | 1,024 | 50-90% (model-dependent) | 30 min+ | None on older models |
| DeepSeek | Automatic | 1,024 | ~90% off | Varies | None |
GPT-5 family models get the deepest cache discount: 90% off input tokens for cache reads. This means GPT-5's effective cached input rate is just $0.125 per million tokens-cheaper than GPT-4.1 Nano's standard input rate.
GPT-4.1 family models get 75% off cached reads. GPT-4o family and o-series models get 50% off cached reads. Model selection interacts with caching in ways most cost analyses ignore.
Why your hit rate is probably lower than you think
Every provider uses prefix matching: the cache key is a hash of the initial N tokens. If those tokens differ between requests-even by a single character-you get a cache miss and pay full price.
This is the trap. The cache is prefix-based, not content-aware. It does not know that your system prompt is "basically the same" as yesterday's. If a timestamp, a user ID, a session variable, or a piece of dynamic working memory lives anywhere inside the cacheable prefix, it busts the cache on every request.
ProjectDiscovery's security agent Neo runs 20 to 40-plus LLM steps per task on top of a 20,000-token system prompt. Their initial cache hit rate was a dismal 7%-because the system prompt contained dynamic working memory that mutated as the agent worked, invalidating the entire cacheable prefix on nearly every step. The fix was a single architectural change: move the dynamic working memory out of the system prompt and place it as a user message at the end of the prompt.
The lesson generalizes: the highest-value caching work is almost never tuning TTLs or breakpoints-it is auditing what dynamic content has crept into the supposedly static prefix.
Beagle in action#engineering, 2:47pmCommon failure modes, roughly in order of how often they appear:
Dynamic content in the prefix. A session ID, a timestamp, a per-user variable embedded early in the system prompt. The fix is always the same: static content first, variable content appended at the end.
Traffic too sparse to stay warm. Anthropic's 5-minute TTL evicts the cache between requests if call volume is low. Switch to the 1-hour TTL for shared system prompts, or pre-warm by firing a request before users arrive.
Prompt below the minimum length. Content needs to be organized into blocks of 1,024 or more tokens. Anything smaller won't cache-that's Anthropic's minimum.
Caching enabled but breakpoints misplaced. The lookback can only find entries that earlier requests already wrote. If a growing conversation pushes your breakpoint 20 or more blocks past the last cache write, the lookback window misses it.
Where this matters most: agents, not chatbots
Prompt caching is not optional for agentic systems. If your agents run more than 3-5 steps, you're leaving significant money on the table. Multi-step tasks are both the most expensive and the most cacheable-the ROI is better than anywhere else in the stack.
The math is straightforward. A conventional chatbot sends a 500-token system prompt once and then exchanges short messages. An agent doing 30 reasoning steps sends the same 20,000-token context-tools, instructions, reference docs-thirty times. Without caching, you pay for all 20,000 tokens thirty times over. With an 84% cache hit rate, you pay full price once and a tenth of full price on the remaining 25 steps.
Cockroach Labs found that re-sent context can be 62% of agent inference bills -and that's the exact slice prompt caching targets.
A teammate like Beagle, running inside Slack or Teams and fielding repeated questions against the same knowledge base, sits squarely in this category-the system prompt and tool definitions are identical across every user request, and the ROI on a warm cache compounds across every team member who triggers it.
How does prompt caching work: common questions
What is prompt caching in LLMs?
Prompt caching reuses precomputed Key-Value attention tensors from a previous request instead of reprocessing identical tokens from scratch. When your new request starts with the same prefix as a cached one, the provider skips that computation and charges a fraction of the standard input token price-typically 10-50% of full cost.
How is prompt caching different from the KV cache?
The KV cache is an in-request optimization that avoids recomputing attention for tokens already generated within a single call. Prompt caching extends this idea across separate API requests: the provider stores those tensors server-side and reuses them when a future request shares the same prefix. Same mechanism, different scope.
Do I need to change my code to use prompt caching?
With OpenAI, no-caching is automatic for prompts over 1,024 tokens and requires no code changes. With Anthropic, you add a cache_control field to the message blocks you want cached.
Cache breakpoints themselves add no cost. You are only charged for cache writes when new content is written, and cache reads when cached content is used.
Why is my cache hit rate low even though caching is enabled?
Almost certainly because dynamic content-a timestamp, a session variable, per-user data-is embedded in the part of your prompt you're trying to cache. Every provider uses prefix matching: the cache key is a hash of the initial tokens. If those tokens differ between requests even by a single character, you get a cache miss. Move all variable content to the end of your prompt, after the stable prefix.
Is prompt caching worth it for short prompts?
Generally not. Both Anthropic and OpenAI require a minimum of 1,024 tokens before caching activates. Short prompts also tend to be cheap to process outright, so the write overhead can outweigh the savings if hit rates are low. The ROI is highest on long, stable prefixes-system prompts, tool definitions, reference documents-reused across many requests within the cache TTL window.