A chatbot with a 5,000-token system prompt serving 10,000 conversations a day is paying for those 5,000 tokens 10,000 times. That's 50 million tokens a day in system prompt costs alone, before a single user message gets processed. Most teams running production LLM applications have no idea this is happening, because the API invoices just show "input tokens." Prompt caching is the structural fix, and it requires either zero code changes or one JSON field, depending on which model provider you use.
Here is what is actually happening, and why the ordering of your prompt matters more than most guides let on.
What the model is doing on every API call
Every LLM API call goes through a "prefill" phase where the model reads your entire prompt and computes attention across all tokens. This is expensive because the computational cost scales quadratically with context length - for a 10,000-token prompt, that means roughly 100 million pairwise comparisons, every single call.
The output of that prefill is a set of matrices - one key matrix and one value matrix per attention layer - that encode what the model "understood" about your input. Providers hold on to these matrices for each prompt for 5-10 minutes after the request is made, and if you send a new request that starts with the same prompt, they reuse the cached key-value tensors rather than recalculating them. That is the entire mechanism. There is no summarization, no compression, no quality change. Cache reads produce identical outputs to fresh processing. The only difference is latency and cost.
The main constraint is prefix matching. Prompt caching works by comparing the beginning of your current prompt against what's already cached. Change anything early in the prompt - a timestamp, a session ID, a user name - and the cache is cold. The model has to recompute from that point forward.
How Anthropic and OpenAI implement it differently
OpenAI and Anthropic do caching very differently from each other. OpenAI does it all automatically for you, attempting to route requests to cached entries when possible.
Automatic caching is enabled for prompts that are 1,024 tokens or longer, with cache hits occurring in increments of 128 tokens. No API change required.
Anthropic's implementation is explicit.
Prompt caching with Anthropic models is explicitly controlled by the developer. Developers mark sections of the prompt as cacheable using the cache_control parameter.
That extra step gives you more control - you can mark up to four separate cache breakpoints - and the discount is steeper:
cache reads cost $0.30 per million tokens versus $3.00 per million fresh on Anthropic.
That is a 90% reduction.
The first request is always a cache write, and cache writes can cost slightly more than a standard input token (on Anthropic, 25% more). The economics only work in your favor once you hit a minimum number of cache reads. As a rough rule: if you reuse a cached prefix more than 2-3 times within the TTL window, you come out ahead.
There is also the time-to-live question. The cache has a TTL that resets with each successful cache hit. During this period, the context in the cache is preserved. If no cache hits occur within the TTL window, your cache expires. Many models support a 5-minute TTL. Anthropic now also offers a 1-hour TTL for newer Claude 4.x models, which avoids constant re-warming for prefixes hit a few times per hour. It only beats the baseline above roughly a 50% hit rate, so reserve it for medium-frequency, high-prefix-size workloads.
A concrete example: an agent loop re-sending 20,000 tokens
Agentic workloads loop the same 20,000-token system prompt across dozens of steps per task. Every one of those tokens was already paid for once and recomputed from scratch on the next call.
Say your agent runs 30 steps per task. Without caching, you pay for 20,000 input tokens 30 times - 600,000 tokens per task run. With Anthropic caching and a 90% discount on cache reads: you pay full price once (the cache write), then 10% of the normal rate on the remaining 29 steps. That one structural change reduces the input cost for those tokens from 600,000 token-equivalents to roughly 78,000.
The catch is prompt structure. In the Codex CLI, system instructions, tool definitions, sandbox configuration, and environment context are kept identical and consistently ordered between requests to preserve long, stable prompt prefixes. The agent loop appends new messages rather than modifying earlier ones when runtime configurations change mid-conversation. If your agent rewrites or shuffles the system block mid-run, every step is a cache miss.
ProjectDiscovery ran into exactly this with their Neo agent. Working memory, runtime context, and per-user variables sat in the middle of the prompt and changed on every step, which led to consistent cache misses on exactly the content that benefits most from caching. After restructuring prompt order and adding explicit breakpoints, they went from single-digit hit rates to 84%.
A teammate like Beagle - which reads from Slack threads, runs multi-step lookups, and re-sends the same tool schemas with every turn - benefits directly from this: stable tool definitions move to the top of the prompt, thread content appends at the bottom, and the majority of input tokens on every step hit the cache rather than the billing counter.
When caching does not help (and what to watch for)
Caching is not free to implement carelessly. Batch processing pipelines that process documents in bursts with gaps longer than 5 minutes will see the cache expire between runs. Every burst starts cold.
Not monitoring cache hit rate is a common failure. Log cache_read_input_tokens vs. cache_creation_input_tokens. If your hit rate is below 50%, your cache is expiring before it's being reused - consider whether your traffic patterns match the TTL, or whether you're invalidating the cache by prepending variable content.
Short prompts don't benefit much either. If your system prompt is 200 tokens, the savings are negligible and you're adding complexity for no meaningful gain. Target prompts of 1,024 tokens or more.
The workloads where caching pays clearly: RAG pipelines injecting large retrieved documents into every query - cache the document context, vary only the user's question. And agentic loops with long tool-use chains where the system prompt and initial context remain constant across many LLM calls in the same session.
The underlying mechanic is simple enough that it is easy to underestimate. You are not changing the model, the output, or the logic of your application. You are changing the order in which you hand content to the model, so that the expensive prefill computation only happens once. Research shows 31% of LLM queries exhibit semantic similarity to previous requests, representing massive inefficiency in deployments without caching infrastructure. For teams already running agents at scale, the question is not whether to implement it - it is why you have not yet.