A security company called ProjectDiscovery was running an AI agent called Neo across 20-to-40-step tasks, each built on a 20,000-token system prompt. Their initial cache hit rate was 7%. After one structural change - moving dynamic working memory out of the prefix and into a trailing user message - it jumped to 84%. That restructuring cut their LLM costs by 59% on actual reported spend, climbing to 66% post-optimization and 70% over the last 10 days of their measurement window. No model changes. No prompt compression. Just order.
That result is the clearest illustration of what prompt caching actually is and why most teams leave it half-working.
What the KV cache actually is
When an LLM processes a prompt, it generates key-value (KV) cache entries in its attention layers - mathematical representations of the relationships between tokens. Normally, the model recomputes this KV cache on every request. Prompt caching stores it so the model can skip that computation on subsequent requests that share the same prefix. The model still generates a fresh response every time; it is the redundant prefill work that gets cut.
To understand why that matters, you need the two-phase shape of LLM inference. LLM inference is divided into two distinct stages: pre-fill, which processes the entire prompt at once to produce the first token and requires heavy compute; and decoding, which generates tokens auto-regressively one at a time. Prompt caching attacks the pre-fill phase specifically. Every LLM request goes through time to first token (TTFT), and it gets worse as prompts get longer - because a long system prompt forces the model to process every token through its attention mechanism before producing any output.
Prompt caching stores the computational state from an LLM's attention layers so the model can skip redundant prefill work on repeated prompt prefixes. The result is lower time-to-first-token and cheaper input costs on every request that hits the cache for a shared prefix.
How each provider implements it
The three major providers landed on caching at different times with structurally different pricing. Anthropic launched prompt caching in public beta on August 14, 2024 and reached general availability on December 17, 2024; OpenAI shipped automatic caching on October 1, 2024; and Google introduced explicit context caching at Google I/O in May 2024, then added zero-setup implicit caching for Gemini 2.5 models on May 8, 2025.
The mechanics differ in ways that matter for real workloads:
| Provider | How you enable it | Minimum prefix | Cache read discount | Write premium | Default TTL |
|---|---|---|---|---|---|
| Anthropic | Explicit cache_control markers |
1,024 tokens (Sonnet/Haiku) | 90% off | 1.25× (5-min) / 2.0× (1-hr) | 5 minutes |
| OpenAI | Automatic on stable prefixes | 1,024 tokens | 50-90% off (model-dependent) | None | Session-scoped |
| Google (Gemini 2.5) | Implicit (no setup needed) | Varies | ~75% off | Storage cost | Configurable |
OpenAI's automatic cache locks onto the longest stable prefix it can find from the start of the request. Anthropic lets you place up to four cache_control markers anywhere in the request - meaning a retrieved document inserted mid-conversation can still cache, and the system prompt and tools can cache as a separate block from a retrieved document below it. For RAG workloads, this is a real structural advantage.
One constraint that catches teams off guard:
on Anthropic specifically, tool definitions must remain byte-identical and in the same order across requests - changing tool_choice, thinking parameters, or an image in the system prompt invalidates downstream cache entries.
Anthropic's cache expires after roughly 5 minutes of inactivity. Each cache hit resets the timer. So an active session - where you are sending messages every minute or two - keeps the cache warm indefinitely. For workloads with gaps between calls, the 1-hour TTL costs twice the write rate but keeps the prefix alive across longer pauses. The 1-hour tier doubles the write price, so breakeven sits above a 50% hit rate - reserve it for traffic with real gaps between invocations.
The one rule that determines hit rate
The core idea is simple: structure your prompts so the static content sits at the top and the dynamic content grows at the bottom. The infrastructure hashes the prefix, stores the KV tensors, and gives you a 90% discount on every subsequent read.
Most teams understand this in principle and break it in practice. The most common failure: one structural change - moving a single dynamic identifier from the middle of the prompt to the end - took the hit rate to 74% and cut the monthly inference bill 59%. The pattern is common: caching is configured correctly in principle, but one dynamic field in the wrong position breaks the entire prefix.
The recommended ordering is:
- Tool definitions - the most stable block; changes rarely
- System prompt - stable within a version
- Reference documents or RAG chunks - stable within a session
- Conversation history - grows per turn but accumulates stably
- Live user query - always at the tail, always dynamic
Any change to a block invalidates that block and everything after it on Anthropic - so dynamic data must live at the very end.
There is a counterintuitive implication worth sitting with: making your prompt slightly longer can actually reduce overall cost. If you have a 900-token prompt you will never get a cache hit. Lengthen it to 1,100 tokens and at a 50% cache rate you save 33% on token costs. At 70% cache rate, you save 55%. Below the provider's minimum, there is nothing to cache regardless of how well-ordered the prompt is.
Beagle in action#product-engineering, 10:22amWhen caching does not help
Caching is not a universal lever. Three workload shapes where it genuinely does not pay:
- Short, mostly-variable requests. A 300-token classification prompt has nothing meaningful to cache.
- Very low volume. A workload that runs ten times a day will pay the cache write cost each time and rarely produce enough reads to recoup it.
- Output-bound workloads. Long-form generation where the output dominates the cost - a 50-token prompt producing 5,000 tokens of output - is unaffected by input caching.
The agent case is where caching earns its keep most clearly. Every time an AI agent takes a step, it sends the entire conversation history back to the LLM - the system instructions, tool definitions, and project context it already processed three turns ago. All of it gets re-read, re-processed, and re-billed on every single turn. A system prompt with 20,000 tokens running over 50 turns means 1 million tokens of redundant computation billed at full price, producing zero new value.
Cache rate naturally improves with task complexity. More steps means more conversation to cache, means a higher hit rate. The architecture compounds in your favor as tasks get longer.
One non-obvious risk most coverage skips: cache hits are measurably faster than cache misses, and an attacker who can measure response latency could potentially infer whether a prompt prefix matches another user's cached prompt. Research has shown attackers can distinguish hits from misses with statistical significance. The practical exploitability depends on your threat model, but it is real. For multi-tenant applications where users share infrastructure, it is worth understanding the latency-timing surface before assuming caches are siloed.
A teammate like Beagle - running inside Slack and triggering LLM calls per message - benefits most from this: the system prompt is always identical, dynamic context always lives at the tail, and a frequently active channel keeps the cache warm without any extra configuration.
Prompt caching: common questions
What is LLM prompt caching?
Prompt caching stores the computational state from an LLM's attention layers so the model can skip redundant prefill work on repeated prompt prefixes. The result is lower time-to-first-token and cheaper input costs on every request that hits the cache for a shared prefix. Output quality is unchanged.
How much does prompt caching reduce costs?
Cache reads bill at 0.10× base input on Anthropic and on OpenAI's newer models; Google's implicit caching delivers a 75% discount. In practice the savings depend entirely on hit rate: a workload at 80% hit rate with a 6,000-token stable prefix will see roughly 59% reduction in billable input tokens.
Why is my prompt cache hit rate low?
The most common cause is dynamic content placed inside the stable prefix. Moving a single dynamic identifier from the middle of the prompt to the end is often enough to take hit rate from near zero to 74%. Other causes: prefix below the provider's 1,024-token minimum, very low request volume that expires the cache between calls, or subtle variation in what looks like identical content (whitespace, different line endings).
Does Anthropic or OpenAI cache better?
For high cache-hit workloads with five or more hits per write cycle, Anthropic's 1.25× write fee is amortized away and the providers tie on read economics. For low cache-hit workloads under three hits per write cycle, OpenAI's no-write-fee structure wins by a few percent.
Anthropic offers more control via explicit cache_control markers and up to four breakpoints per request; OpenAI requires zero configuration.
Does prompt caching affect model output?
No. The model still generates a fresh response every time; it is the redundant prefill work that gets cut. Caching the KV state of a prefix changes inference cost and latency, not the probability distribution the model samples from.