ProjectDiscovery's security agent Neo runs 26 steps per task and fires 40 tool calls per run, with a system prompt stretching past 20,000 tokens. Every step re-sends the whole thing - system prompt, tool definitions, conversation history - to the model. Before they touched their prompt structure, their cache hit rate was 7%. One architectural change later, it was 84%, and their LLM bill dropped 59%.
That number is not a rounding error. It points at something real: prompt caching is probably the highest-leverage cost cut available to teams running LLM workloads at any volume, and most teams either don't use it or accidentally break it. Understanding why requires knowing what the API is actually storing.
What the model computes - and what caching skips
Every time you send a prompt to an LLM, the model processes each token through its attention mechanism and builds what's called a KV cache: a matrix of key-value pairs representing the relationships between tokens. When an LLM processes a prompt, it generates key-value cache entries in its attention layers - and normally, the model recomputes this KV cache on every request.
Prompt caching stores it so the model can skip that computation on subsequent requests that share the same prefix. The model still generates a fresh response every time; it's the redundant prefill work that gets cut.
The part that gets cached is always the beginning of the prompt - the prefix. It stores and reuses the initial, unchanging part of a prompt so that large language models don't have to process it again on every request. Which means the order of your prompt matters enormously. A timestamp, a session ID, or a user name sitting near the top of your system prompt will silently bust the cache on every single call.
Prompt caching stores the computational state from an LLM's attention layers so the model can skip redundant prefill work on repeated prompt prefixes. The result: lower time-to-first-token (TTFT) and cheaper input costs on every request that hits the cache for a shared prefix.
How OpenAI and Anthropic implement this differently
OpenAI and Anthropic do caching very differently from each other. OpenAI does it all automatically for you, attempting to route requests to cached entries when possible.
Here's the practical breakdown:
| OpenAI | Anthropic | |
|---|---|---|
| Activation | Automatic, no code changes | Manual cache_control breakpoints |
| Cache read discount | 50% off input tokens | 90% off input tokens |
| Write premium | None | 1.25× standard (5-min TTL) or 2.0× (1-hour TTL) |
| Hit rate guarantee | ~50% (best-effort) | 100% when prefix matches |
| Minimum prefix | 1,024 tokens | 1,024 tokens (Sonnet/Opus); 2,048 (Haiku) |
| Default TTL | 5-10 minutes | 5 minutes; 1-hour option available |
OpenAI prompt caching is fully automatic - zero code changes, 50% cost discount on cached tokens, roughly 50% hit rate (best effort, not guaranteed). Anthropic prompt caching is manual - you set cache_control breakpoints, get a 90% cost discount on cache reads, and a 100% guaranteed hit rate when configured correctly.
The asymmetry between write and read costs on Anthropic deserves a closer look. With the 5-minute TTL (1.25× write, 0.10× read), you save the write premium back after one cache hit. Hit once, you've already broken even. Every hit after that is pure savings.
Implementation details such as minimum token thresholds (typically 1,024-4,096 tokens depending on model), TTL durations (ranging from 5 minutes to 24 hours), and pricing structures vary across providers and are subject to change. Worth checking the pricing docs directly before building your cost model.
Why your cache hit rate is probably low (and the fix is structural)
Position is everything - a change near the beginning of your prompt invalidates the front and everything after it, because the match stops at the first difference. A change near the back invalidates only the back. Identical-but-unreachable content is worthless: if two requests differ at token 50, the 3,000 identical tokens sitting at position 51 onward cache nothing.
This is the exact trap ProjectDiscovery fell into. Their security agent Neo runs 20 to 40-plus LLM steps per task on top of a 20,000-token system prompt. Their initial cache hit rate was a dismal 7% - because the system prompt contained dynamic working memory that mutated as the agent worked, invalidating the entire cacheable prefix on nearly every step.
The fix was a single architectural change: move the dynamic working memory out of the system prompt and place it as a user message at the end of the prompt. That one relocation raised the cache hit rate to 84% overnight and cut overall LLM cost by 59% - peaking at 70% in the final 10 days of the engagement, with 9.8 billion tokens ultimately served from cache.
Order content most-to-least stable: tool definitions, then system prompt, then reference docs, then conversation history, then the live user query. Any change to a block invalidates that block and everything after it on Anthropic - so dynamic data must live at the very end.
A second, less obvious failure mode appears in agentic loops specifically. Agentic workloads are the worst-case user of this mechanism. An agent's loop is: think, act, wait. It sends a request, then runs a tool - a build, a test suite, a deployment, a wait for human approval - that takes minutes, and only then sends the follow-up that would have reused the now-large conversation prefix.
Provider caches expire in minutes: Anthropic's default TTL is five minutes, OpenAI clears entries after "5-10 minutes of inactivity." The pause outlives the cache, and the follow-up pays full price and full latency.
Tool definitions are another quiet invalidation trigger. A February 2026 evaluation of prompt caching for agentic workloads states the rule plainly: when tool definitions are in the prompt, "any change to the available tool set invalidates the cached prefix" and recommends a fixed tool set as the mitigation.
Beagle in action#engineering, Tuesday 10:02amThe non-obvious cost: staleness and the 5-minute cliff
There is a real risk that coverage on prompt caching almost never mentions: stale instructions getting served from cache. A team updates approval logic or redaction rules, but an old cache key keeps reusing outdated instructions. The agent now behaves consistently wrong, which is worse than being randomly wrong.
Fix: version every stable prompt asset and force invalidation on important policy changes.
The same applies to tool schemas:
the prompt still describes create_ticket(priority, owner) but the API now expects create_ticket(severity, assignee). Cached tool instructions silently fall out of sync. Fix: tie cache keys to tool schema versions, not just workflow names.
There is also a security angle worth knowing. Cache hits are measurably faster than cache misses. An attacker who can measure response latency could potentially infer whether a prompt prefix matches another user's cached prompt, or patterns that could enable prompt reconstruction through binary search. Research has shown attackers can distinguish hits vs. misses with statistical significance (p < 10⁻⁸). For most internal tooling this is theoretical, but for customer-facing products it belongs in your threat model.
The arithmetic is blunt. For a 10,000-token system prompt at 2,000 requests per day with a 75% hit rate on Claude Sonnet, the monthly saving is roughly $1,215 - from one structural decision about where dynamic data lives in the prompt.
Prompt caching: common questions
What is prompt caching in LLMs?
Prompt caching stores the computational state from an LLM's attention layers so the model can skip redundant prefill work on repeated prompt prefixes. The result: lower time-to-first-token and cheaper input costs on every request that hits the cache for a shared prefix. The model output is always freshly generated - only the input processing is reused.
How much does prompt caching actually save?
Anthropic's prompt caching reduces costs by up to 90% and latency by up to 85% for long prompts. OpenAI achieves 50% cost reduction with automatic caching enabled by default. Real-world savings depend on hit rate, which depends almost entirely on whether dynamic data is kept out of the cacheable prefix.
Does OpenAI prompt caching require any code changes?
No.
OpenAI offers automatic prompt caching on GPT-4o and newer models, where caching activates automatically for prompts exceeding a minimum token threshold, with cache hits occurring only for exact prefix matches.
You see the savings in the cached_tokens field of the API response.
Why does adding one tool to my agent break the cache?
When tool definitions are in the prompt, any change to the available tool set invalidates the cached prefix. The prefix match stops at the first changed token, so a reordered or added tool definition - even a small one - will bust the cache for everything that follows it. Keep tool definitions in a fixed order and add new tools at the end.
How long does a cached prompt stay cached?
TTL durations range from 5 minutes to 24 hours depending on the provider and tier.
By default, cached prefixes stay in GPU VRAM for 5-10 minutes of inactivity. The extended 24-hour retention offloads KV tensors to GPU-local storage (SSDs attached to GPU nodes) when idle, loading them back into VRAM on a cache hit. For agentic workloads where tool execution takes several minutes, the default 5-minute TTL frequently expires between steps - a strong argument for the 1-hour tier.