Paste a 200-page contract into Claude and ask a question. The model answers correctly. Paste a second 200-page contract and ask which one contained a specific clause. Now you have a problem - not because the window is too small, but because of something structural that happens inside it that the token-count headline doesn't tell you.
The LLM context window is probably the single concept most worth understanding if you work with AI tools daily. It explains why models go wrong in ways that feel random, why your costs compound over a long conversation, and why a million-token window doesn't mean a million tokens of perfect memory.
What a context window actually is
The context window defines the number of tokens an LLM can process at once, influencing its ability to maintain context and coherence. A token is roughly three-quarters of an English word, so a 128,000-token window holds around 96,000 words - about a 300-page book.
But here is the part most descriptions skip: the model doesn't remember anything. It's not storing your conversation the way a database would. Instead, it's rereading the entire conversation from the beginning every single time you send a message. Every turn, the application sends the full history back to the model as input, the model re-processes all of it, and only then generates its reply.
This has a direct cost consequence. In one logged Claude session, a 14-token user question at the start cost about $0.0018 to process; by the 260th exchange, the same 14-token question cost roughly $2.41 - a 1,339× increase purely due to accumulated history. The input being processed grew with every turn, not just the new message.
The computational reason this is expensive: the computational costs grow quadratically as the number of tokens increases. This occurs because the self-attention mechanism in Transformer architectures calculates attention scores between all token pairs, resulting in time and memory complexity of O(n²).
The KV cache: what makes repeated context tolerable
If every generation step recomputed attention for every token from scratch, even moderately long conversations would be unusably slow. The KV cache is the optimization that prevents this.
During the attention calculation, each token in the sequence is projected into three vectors: a query, a key, and a value. The query is consumed immediately. The KV cache is implemented by caching per-layer attention keys and values from previous tokens during decoding and reusing them for subsequent tokens.
Without KV caching, these matrices are recomputed for all previous tokens at every step, leading to unwanted resource waste. The KV cache stores the keys and values from earlier tokens and, at each new step, only computes them for the freshly generated token and retrieves the rest from the cache.
Concretely: when you write message 50 in a conversation, the model doesn't re-run math on messages 1-49. It just reads their already-computed key and value vectors from memory. This is also the foundation that makes prompt caching possible as an API feature.
Prompt caching optimizes API usage by allowing resuming from specific prefixes in your prompts, significantly reducing processing time and costs for repetitive tasks or prompts with consistent elements.
Cache hits require an exact, repeated prefix match and work automatically for prompts containing 1,024 tokens or more, with cache hits occurring in increments of 128 tokens. One practical implication: the cached prefix must be exactly identical between calls - even a 1-bit difference triggers a miss. If you put the user's name or a timestamp at the top of your system prompt, you're breaking your own cache on every request.
The blind spot in the middle
A larger context window sounds like a straightforward improvement. The uncomfortable research finding is that it isn't quite that simple.
LLM performance on multi-document question answering and key-value retrieval follows a U-shaped function of information position: accuracy is highest when relevant information appears at the beginning or end of the input context and degrades by more than 30% when relevant information is positioned in the middle.
This finding replicated across six model families including GPT-3.5-Turbo, GPT-4, and Claude 1.3.
The cause is architectural, not accidental. The root cause lies in the attention mechanisms and positional encodings used by transformer-based models. Rotary Position Embedding (RoPE), commonly used in modern LLMs, introduces a long-term decay effect that causes models to prioritize tokens at the beginning and end of sequences while de-emphasizing middle content.
Newer models have reduced the severity but not eliminated it. A study by Chroma testing 18 frontier models found that performance degrades at every context length increment, not just near the limit. A model with a 1M token window still exhibits measurable degradation at 50K tokens.
This reframes the context window arms race. A 1M token window doesn't work like 1M tokens of perfect memory. Token count is the ceiling, not a reliability guarantee.
What this means for how you structure inputs
Understanding the mechanism changes how you should use these tools.
Put the most important content first or last, not buried in the middle. If you're feeding a model 10 documents and asking a question, put the most relevant document at position 1 or position 10, not position 5. The model's attention will follow its architectural bias regardless of what the content says.
Keep shared prefixes stable. System prompts, document context, and tool definitions that don't change between turns belong at the very start of the prompt, unchanged. Cache stable, reusable content like system instructions, background information, large contexts, or frequent tool definitions. Place cached content at the prompt's beginning for best performance.
Treat long multi-turn conversations as a growing cost. Each new message doesn't just add its own tokens - it re-presents the entire history. Everything the model sees and reasons over lives in one shared context buffer. User messages, assistant replies, system instructions, uploaded files, images, and tool outputs all consume tokens from the same window. A Beagle-style assistant in a busy Slack channel, for instance, can trim conversation history that's no longer relevant rather than appending indefinitely.
Don't assume a large window eliminates the need for retrieval. For long documents, the window determines whether a report or contract can be analysed in a single pass. When a document exceeds the limit, tools either chunk it behind the scenes or use retrieval to pull only relevant sections at query time. Even when a document fits, chunking and ranking may still surface the right passage more reliably than relying on the model's attention to find it at position 487,000.
The context window is not a hard disk. It's closer to a whiteboard that gets re-read - at full cost - every time someone enters the room. Understanding that changes how you write prompts, design agents, and interpret unexpected model behavior. The token count on the marketing page is the size of the whiteboard. Whether the model reads the right part of it is a different question entirely.