By the time an agent finishes a 12-turn research session, it may have sent the same 10,000-token system prompt to the model a dozen separate times - and paid full price every single time. That is the problem prompt caching exists to fix, and it is quietly one of the most consequential architectural decisions in any production LLM deployment.
What LLM Prompt Caching Actually Stores
The phrase "prompt caching" sounds like it saves and replays the model's answer. It doesn't. When an LLM processes your prompt, it generates key-value (KV) cache entries in its attention layers - mathematical representations of the relationships between tokens. Normally the model recomputes this KV cache on every request. Prompt caching stores it so the model can skip that computation on subsequent requests that share the same prefix. The model still generates a fresh response every time; it's the redundant prefill work that gets cut.
That distinction matters. Cache reads produce identical outputs to fresh processing. Nothing is being shortcut in the reasoning. What changes is the amount of GPU time burned before the first output token appears.
Every LLM request goes through two latency phases: time to first token (TTFT), which measures how long the model takes to start responding, and time to last token (TTLT), which captures the full generation time. Both get worse as your prompts get longer. A long system prompt increases TTFT because the model processes every token through its attention mechanism before producing any output.
When the model skips computing KV states for cached tokens, it begins generating output much sooner. For an agent with a 5,000-token system prompt, a cache hit means the model starts working on just the new 100-token user message almost immediately - instead of wading through the full 5,100 tokens first. In practice, this translates to 2-5× faster first-token latency on cache hits.
Why Prompt Order Determines Whether Caching Works
The main constraint is prefix matching. Prompt caching works by comparing the beginning of your current prompt against what's already cached. The moment the content diverges from the cached version, the match breaks and you pay full price for everything that follows.
This has a concrete implication for how you structure requests. Place static content like instructions and examples at the beginning of your prompt, and put variable content, such as user-specific information, at the end. This also applies to images and tools, which must be identical between requests.
A timestamp injected into the system prompt at the top of the request invalidates the entire cache. A request ID dropped into the tool list breaks the tool-list cache. The single most common mistake is putting a variable string before the long stable content.
A well-ordered prompt for an AI assistant in Slack might look like this: the system persona and instructions first (stable), then the full tool schema (stable per deployment), then any retrieved documents relevant to this session (mostly stable), and finally the user's actual message (changes every time). Everything before the user message is a candidate for the cache. The user message itself is not.
Beagle in action#engineering, 11:02amHow Anthropic and OpenAI Handle It Differently
The two leading providers have made opposite design choices about who controls the cache.
OpenAI offers automatic prompt caching on GPT-4o and newer models, where caching activates automatically for prompts exceeding a minimum token threshold, with cache hits occurring only for exact prefix matches. Anthropic provides developer-controlled caching through explicit cache breakpoints, allowing users to specify which portions of their prompt should be cached, with configurable time-to-live (TTL) options.
In practice: OpenAI's design assumes you'd rather not think about it. If your traffic is high-frequency and your prompts are large and stable, Anthropic's 90% discount will dominate the comparison; if your traffic is sporadic or your prompts evolve frequently, OpenAI's automatic 50% with no write premium is the cleaner choice.
The pricing gap is real. When a cached prefix is read, providers charge a fraction of the normal input token price - Anthropic charges about 10% (a 90% discount), OpenAI charges 50% off, and Google charges roughly 25% of the normal rate. But Anthropic's model charges a write premium - the question is whether the discount on cache reads outweighs the premium on the first write. With the 5-minute TTL (1.25× write, 0.10× read), you save the write premium back after one cache hit. Hit once, you've already broken even. Every hit after that is pure savings.
The agentic case is where the numbers get significant. Agentic workloads loop the same 20,000-token system prompt across dozens of steps per task; retrieval pipelines re-send the same document corpus on every query; chat products replay an entire conversation history each turn. Every one of those tokens was already paid for once and recomputed from scratch on the next call. Caching reclaims that spend - but only if the cacheable block is genuinely stable.
A January 2026 paper from arXiv measured this directly across 500 agent sessions. Prompt caching reduces API costs by 41-80% and improves time to first token by 13-31% across providers. Strategic prompt cache block control, such as placing dynamic content at the end of the system prompt, provides more consistent benefits than naive full context caching.
When Prompt Caching Stops Helping
Caching is not free, and there are workloads where it actively adds cost. Writing to cache costs 25% more than a normal input token on some providers. The savings come from repeated reads. If you're only calling an endpoint once, caching adds cost, not reduces it. Caching pays off when the same prefix is used at least 2-3 times.
Agent loops with live data in the context are a specific trap. Unlike static question-answering scenarios where prompts are largely predetermined, agentic workloads feature dynamic, session-specific content that accumulates unpredictably. Tool results often contain user-specific data that will not benefit other sessions, and the interleaving of static system prompts with dynamic tool outputs complicates cache reuse.
The practical rule: the cached content must be a contiguous block at the beginning of your prompt, identical in every request. Static content at the start - system instructions, tool definitions, document context - gets cached. Variable content at the end, the user's specific question, doesn't.
A teammate like Beagle benefits directly from this architecture. Every request it handles in Slack shares a large stable prefix - the same persona, the same tool definitions, the same workspace context - with only the specific question changing at the end. That structure is exactly what caches well, which is why Beagle's integrations are built to keep static context front-loaded.
The TTL is the last variable to get right. Minimum token thresholds (typically 1,024-4,096 tokens depending on model), TTL durations (ranging from 5 minutes to 24 hours), and pricing structures vary across providers and are subject to change. If requests in your workflow are spaced more than five minutes apart - a human approval step, a slow external API call - you need the one-hour TTL tier or your cache expires between turns and you pay for the write without collecting the read discount.
The cost arithmetic is not complicated once you see it. The hard part is keeping your prompts ordered correctly, which most teams only audit after they look at a billing dashboard and notice the caching discount is smaller than expected.