A 5,000-token system prompt sent to 10,000 conversations a day adds up to 50 million tokens a day in system prompt costs alone - before a single user message is processed. Most teams pay that bill on every single request, every single day, for the same static text. Prompt caching is the mechanism that makes those repeated tokens near-free - and most production deployments are barely using it.
What prompt caching actually stores
Prompt caching stores the computed key-value tensors behind a repeated prompt prefix so that the static portion of every request - tool definitions, system prompt, and reference documents - bills at up to 90% off, with the model producing byte-identical output. No distillation, no quantization, no quality trade-off. Just a structural change to how you order a prompt.
The underlying mechanic is easier to follow than it sounds. Modern generative LLMs are decoder-only and use causal (masked) self-attention. Tokens can only look backwards. Information only flows left to right. That one-way structure is what makes caching safe. Because of this, we are not caching a fragile, easily broken global attention matrix. We are caching the Key (K) and Value (V) tensors of the prefix sequence. The K and V states of the early context are calculated once and never change, regardless of what user prompt you append at the end. The model simply takes your new tokens, generates their Queries, and runs them against the heavily cached Keys of the past.
The result: prompt caching stores the computational state from an LLM's attention layers so the model can skip redundant prefill work on repeated prompt prefixes. Lower time-to-first-token and cheaper input costs on every request that hits the cache.
LLM outputs are dynamic and constantly changing, so caching the input is what makes sense. The model still generates fresh output every time. It just skips the expensive work of re-reading your boilerplate.
How the three major providers differ
The three major providers landed on caching at different times and with structurally different pricing. Anthropic launched prompt caching in public beta on August 14, 2024 and reached general availability on December 17, 2024; OpenAI shipped automatic caching on October 1, 2024; and Google introduced explicit context caching at Google I/O in May 2024, then added zero-setup implicit caching for Gemini 2.5 models on May 8, 2025.
The implementation philosophies diverge significantly:
| Provider | Setup required | Cache hit discount | Write cost | Min. tokens | TTL |
|---|---|---|---|---|---|
| Anthropic | Explicit cache_control markers |
90% off | +25% (5-min) / +100% (1-hr) | 1,024-4,096 (model-dependent) | 5 min or 1 hour |
| OpenAI | Automatic | 50% off (up to 90% on newer models) | None | 1,024 | ~5-10 min |
| Implicit or explicit | 75-90% (model-dependent) | Storage fee (explicit) | 32K (explicit) | Per-hour |
OpenAI offers automatic prompt caching on GPT-4o and newer models, where caching activates automatically for prompts exceeding a minimum token threshold, with cache hits occurring only for exact prefix matches.
Anthropic provides developer-controlled caching through explicit cache breakpoints, allowing users to specify which portions of their prompt should be cached, with configurable time-to-live options.
Google offers both implicit caching, which activates automatically with no guaranteed cost savings, and explicit context caching, where developers create and reference caches with guaranteed discounts.
On minimum thresholds: minimum token thresholds typically range from 1,024 to 4,096 tokens depending on the model, and TTL durations range from 5 minutes to 24 hours.
Caching is silently skipped when input is below the provider's minimum - no error is returned.
This trips people up constantly. If you're seeing cache_read_input_tokens: 0 on every Anthropic request, your prefix is probably too short.
The prompt-ordering mistake that kills your cache hit rate
The most important practical rule: static content goes first, dynamic content goes last. But the subtlety is in what counts as "dynamic."
On Anthropic specifically, the discipline is stricter than "put dynamic data last." Tool definitions must remain byte-identical and in the same order across requests; changing tool_choice, thinking parameters, or an image in the system prompt invalidates downstream cache entries.
The sharpest case study in the wild is from ProjectDiscovery. Their security agent Neo runs 20 to 40-plus LLM steps per task on top of a 20,000-token system prompt. Their initial cache hit rate was a dismal 7% - because the system prompt contained dynamic working memory that mutated as the agent worked, invalidating the entire cacheable prefix on nearly every step. Once they separated static instructions from dynamic state, their cache hit rate climbed to 84%, cutting total LLM spend by 59-70%.
A research evaluation of prompt caching across OpenAI, Anthropic, and Google found it reduces API costs by 45-80% and improves time to first token by 13-31% across providers - but only when dynamic content is placed at the end of the system prompt and excluded from cached blocks. Naive full-context caching can paradoxically increase latency.
Beagle in action#ai-infra, 2:47pmAgentic workloads: where caching earns the most, and fails the hardest
Agentic workloads loop the same 20,000-token system prompt across dozens of steps per task; retrieval pipelines re-send the same document corpus on every query; chat products replay an entire conversation history each turn. That repetition is exactly where caching pays off most - but it's also where the pitfalls are sharpest.
If your workflow is dense and continuous, short cache windows work fine. If your workflow pauses - waiting for builds, human review, or external processes - longer retention becomes critical. This matters especially for orchestrator agents. An orchestrator's job is to coordinate: dispatch workers, wait for results, decide what comes next. That pattern is inherently stop-and-go. An orchestrator that waits three minutes for a build result needs a cache that survives three minutes of silence.
If requests are spaced more than 5 minutes apart (for Anthropic's short TTL) or more than the provider window allows, caches expire between requests, making every request a cold start with cache write costs but no read benefits. If your average request interval exceeds 5 minutes on Anthropic, the 5-minute cache will frequently miss - the 1-hour cache costs more to write but may hit more often.
One structural insight that most coverage misses: Anthropic allows up to four cache breakpoints per request, and caches are processed in the order of tools, system, then messages. That ordering matters. Tool definitions that sit before a cache breakpoint must be frozen. Any variation - even changing their order - is a cache miss.
The practical prompt structure, from most stable to least:
- Tool definitions - frozen, in fixed order, cached first
- System instructions - static rules and persona, cached
- Reference documents - retrieved corpus, cached if the query set is predictable
- Conversation history - append-only, partially cacheable
- Current user query / tool result - always dynamic, always last
A teammate like Beagle, which reads and writes inside Slack threads, benefits from exactly this shape: the instruction layer is stable, the conversation is append-only, and only the final user message changes each turn.
Prompt caching in Slack-based AI tools: common questions
What is prompt caching in LLMs?
Prompt caching stores the computed attention key-value tensors for repeated prompt prefixes, so the model skips re-processing static content like system instructions or tool definitions on subsequent requests. The output is mathematically identical to a fresh request; only the billing and latency change. Savings reach 50-90% on input tokens depending on provider.
Does prompt caching change the model's output?
No. The model processes cached tokens identically to fresh tokens. It simply skips the re-computation of attention for those tokens. The output is mathematically equivalent.
What breaks a prompt cache?
Any modification to content before the cache boundary causes a miss. Common culprits: inserting a timestamp or user name into the system prompt, reordering tool definitions, switching models mid-session, or letting working memory accumulate inside a static prefix block. Keep everything before the cache breakpoint byte-identical across requests.
Which provider's prompt caching is best for agentic workflows?
It depends on your request cadence. For most developers, OpenAI's automatic caching is the right starting point - it requires no code changes and immediately applies to any prompt over 1,024 tokens. If you're at significant scale (500K+ tokens/day in static context), Anthropic's 90% discount justifies the implementation overhead. For orchestrators with pauses longer than five minutes, use Anthropic's 1-hour TTL option.
How do I verify my cache is actually working?
On OpenAI, check the usage object in the response for cached_tokens. On Anthropic, look for cache_creation_input_tokens and cache_read_input_tokens in the usage block. If both are 0 on every request, the cache isn't activating - usually because your prompt is below the minimum token threshold or dynamic content appears before the cacheable section.