LLM Prompt Caching: What the KV Cache Actually Stores

One company raised their cache hit rate from 7% to 84% and cut their total LLM spend by 59-70%. They weren't doing anything exotic - they reorganized which tokens appeared first in their prompts.

That's the whole trick with prompt caching, and it's worth understanding precisely why it works. The mechanism is specific, the failure modes are silent, and the savings compound hard in agentic workflows where the same system prompt gets re-sent on every loop iteration.

What the KV cache actually stores

When an LLM processes your prompt, it generates key-value (KV) cache entries in its attention layers - mathematical representations of the relationships between tokens. Normally, the model recomputes this KV cache on every request. Prompt caching stores it so the model can skip that computation on subsequent requests that share the same prefix.

To be concrete about what that means in hardware terms: for each token in the context, the KV cache must store two vectors per attention layer - one for keys, one for values - across every attention head. For a model like LLaMA-13B, that works out to roughly 800 KB per token, meaning a single 2,048-token sequence requires around 1.6 GB of KV cache. Production models are larger, but the principle is identical. By reusing the stored K and V matrices, the model reduces inference time complexity from O(n²) to O(n), where n is the sequence length - attention is computed only for newly generated tokens.

What the provider caches across API requests is a serialized snapshot of those tensors for a given prefix. By default, cached prefixes stay in GPU VRAM for 5-10 minutes of inactivity. Extended 24-hour retention offloads KV tensors to GPU-local storage (SSDs attached to GPU nodes), loading them back into VRAM on a cache hit.

Crucially: the model still generates a fresh response every time - it's the redundant prefill work that gets cut.

The model processes cached tokens identically to fresh tokens; it simply skips the re-computation of attention for them. The output is mathematically equivalent.

How prefix matching works - and where it silently breaks

Prompt caching stores the computational state from an LLM's attention layers so the model can skip redundant prefill work on repeated prompt prefixes. The result: lower time-to-first-token (TTFT) and cheaper input costs on every request that hits the cache.

The constraint is strict. Any change to the cached prefix, even a single character, invalidates the cache completely. There's no partial cache invalidation.

A timestamp injected into the system prompt at the top of the request invalidates the entire cache. This is the most common silent failure: a team enables caching, then discovers their logging middleware is writing a request ID into the system prompt header. Hit rate: zero.

The ordering rule follows directly from how prefix matching works: put everything stable first, everything variable last. The order that produces the best hit rate is system prompt (most stable), then tool definitions, then long static context like recurring corpus excerpts, then slowly changing conversation history, then the current user message (most variable).

Anything ordered after a variable element does not cache.

Anthropic vs OpenAI: two different models, same 90% ceiling

The two providers have converged on discount level but not on implementation.

The cached-input discount race is essentially over. Both Anthropic and OpenAI's newest flagships now bill cached reads at 10% of the standard input rate - a 90% discount that started as Anthropic's signature.

OpenAI's automatic cache locks onto the longest stable prefix it can find from the start of the request. Anthropic lets you place up to four cache_control markers anywhere in the request - meaning a retrieved document inserted mid-conversation can still cache, and the system prompt and tools can cache as a separate block from a retrieved document below it. For RAG workloads, this is a real structural advantage.

The write premium on Anthropic deserves a direct look: with the 5-minute TTL (1.25× write, 0.10× read), you save the write premium back after one cache hit. Hit once, you've already broken even. Every hit after that is pure savings.

Why agent loops are where caching matters most

A chatbot with a 2,000-token system prompt benefits from caching. An agent loop running 10 steps with that same system prompt benefits ten times as much - and the math gets steep fast.

A research agent running 8 steps with an average context of 5,000 tokens per step processes 40,000 input tokens per query. With caching, you might pay for only 8,000 - the truly new tokens at each step. That's an 80% reduction.

Cache reads bill at 0.1× the input rate - in a worked 10-turn agent loop, that turns $0.51 of input spend into $0.14 per task. The fixed-cost portion - system prompt, tool definitions - gets amortized across every turn. Only the new context (tool outputs, user messages, model reasoning) pays full price.

The growing conversation history is the dominant cost driver in multi-step loops, and each new tool output or reasoning trace is unique per iteration, so it cannot be cached. This is an important boundary to hold in mind: caching is not a silver bullet for long agent loops. It kills the fixed costs. The variable accumulation - which grows quadratically with loop length - still has to be managed separately, through summarization or context pruning.

One non-obvious wrinkle specific to agentic frameworks: an agent loop re-sends the same system prompt, the same MCP tool definitions, and a growing conversation prefix on every single turn - exactly the shape the cache was built for. If your MCP tool list changes between turns (say, tools are conditionally added mid-loop), the cache breaks at that point. Keep the tool list stable for the life of a session, even if some tools aren't used on every turn.

How to actually get a high hit rate

A high theoretical discount and a low real discount are completely compatible - if the cache never hits.

• Put dynamic content last. User messages, retrieved chunks unique to this query, timestamps - all go at the end. Stable content (system prompt, tool schemas, reused documents) goes first.

• Freeze system prompt updates. Plan system prompt updates as deployment events: update during off-peak hours and expect a brief period of full-price requests while the new prefix warms up.

• Use Anthropic's pre-warm endpoint. Anthropic supports cache pre-warming: sending a request with max_tokens: 0 to prefill the prompt and write the cache entry without generating any output. The model processes your system prompt or tool definitions, saves the KV tensors to the cache, and returns immediately with zero output tokens. The first real request from a user hits a warm cache and gets the latency benefit immediately.

• Measure before assuming. Check cache_read_input_tokens in your API response (Anthropic) or cached_tokens in usage data (OpenAI). If your hit rate is below 60% on a stable-prefix workload, something in your prompt order is wrong.

• Watch the minimum threshold. Below a per-model minimum, nothing caches - silently. The request succeeds; you just paid full price. On Bedrock, some Claude models require 4,096 tokens at the checkpoint, not 1,024.

A teammate like Beagle - running inside Slack and Teams against a stable system prompt and a fixed tool set - sits in exactly the pattern where caching earns its keep: the same large prompt prefix on every call, high request frequency, near-zero variation in the static block.

LLM prompt caching: common questions

What is prompt caching in LLMs?

Prompt caching stores the KV tensor state from an LLM's attention layers for a given prompt prefix. When a subsequent request begins with the same prefix, the model skips recomputing that state and reads from cache instead. The result is lower latency and discounted input token pricing - up to 90% off on Anthropic and OpenAI flagship models.

Does prompt caching change the model's output?

No. The model processes cached tokens identically to fresh tokens - it simply skips the re-computation of attention for those tokens. The output is mathematically equivalent. Caching is a compute optimization, not an approximation.

How long does a prompt cache last?

Anthropic caches expire after 5 minutes of inactivity, or 1 hour with extended caching configured. OpenAI's cache duration is typically 5-10 minutes. After expiration, the next request writes a new cache entry at normal or write-premium pricing, then subsequent requests hit the refreshed cache. OpenAI has extended retention to 24 hours for GPT-4.1 and the GPT-5.1 series.

When does prompt caching not help?

Output-bound workloads - long-form generation where the output dominates the cost - are unaffected by input caching. It also doesn't help when your prompt prefix changes on every request (unique user IDs, timestamps, or per-query retrieved context inserted near the top). And in agent loops, caching only addresses the fixed system prompt portion of each call - the growing conversation history is the dominant cost driver in multi-step loops, and each new tool output or reasoning trace is unique per iteration, so it cannot be cached.

Which provider has better prompt caching: Anthropic or OpenAI?

Both now match on the read discount (90% off). The difference is implementation: Anthropic's prompt caching is more powerful but requires explicit implementation - you control exactly what gets cached using cache_control markers in your API requests. OpenAI's is automatic and zero-config, which wins on simplicity. For RAG pipelines with mid-prompt document insertion, Anthropic's multi-breakpoint system has a structural edge. For standard chat with a stable system prompt, OpenAI's automatic approach is simpler and equally priced on current models.