You paste a 40-page contract into Claude, ask a specific question about the indemnity clause on page 28, and get a confident wrong answer. The clause is right there in the window. The model just didn't use it well.
That failure has a name, a mechanism, and a fix - none of which vendors put in their spec sheets.
How the context window actually works
The context window is the total text a model can hold in working memory for a single inference: your prompt, any system instructions, retrieved documents, conversation history, and the model's own reply all share the same fixed space. Everything has to fit inside it; go over the limit and the model either refuses the request or silently drops the oldest content.
The engine underneath is the attention mechanism. Before generating each new token of its response, the model compares it against every other token currently in the context window - this gives it the ability to connect ideas across long stretches of text, but it's also the source of its most important limitations.
That comparison is expensive. The computational cost scales quadratically with sequence length - doubling the context window roughly quadruples the computation for the attention layers. This is why larger context windows are expensive. Put another way: a 100k-token prompt doesn't cost twice as much as a 50k-token prompt to process. It costs roughly four times as much in raw compute.
The model also stores intermediate computation as it reads your input. When generating a reply, the model doesn't reprocess your entire conversation from scratch each time - it maintains a KV cache, the computed keys and values from previous tokens. That cache is memory-hungry. For context windows beyond approximately 32K tokens, the KV cache consumes more memory than the model itself, fundamentally changing deployment requirements. This is why serving Claude at 100K context is categorically different, in infrastructure terms, from serving it at 4K context. The model has not changed. The hardware costs have multiplied.
The spec sheet number is the wrong number
In July 2026, mainstream flagship models process about 1 million tokens - roughly 750,000 words or 1,500 pages - including GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8, and Claude Sonnet 5. That race to bigger numbers obscures the more important figure: effective context, which is how much of that window the model actually uses reliably.
NVIDIA's RULER benchmark puts usable context at 50-65 percent of advertised, and Adobe's NoLiMa research found most models drop below half their short-context score by 32K tokens.
The specific failure mode has a name: lost in the middle. Stanford and UC Berkeley researchers first documented this in 2023: models attend well to the beginning and end of context but poorly to the middle. Accuracy dropped by more than 30% when relevant information was placed in middle positions compared to positions 1 or 20, in multi-document question answering.
The mechanism is attention dilution. As context grows, the model's finite attention budget gets spread thinner across more tokens. Information that was highly attended at 1,000 tokens may be functionally ignored at 100,000 tokens. The indemnity clause on page 28 isn't lost because the model can't read it - it's lost because every other sentence in the document is competing for the same limited pool of attention weight.
A 2025 research study by Chroma tested 18 of the most powerful language models, including GPT-4.1, Claude, and Gemini, and found that every single one performed worse as the amount of input grew. The degradation wasn't minor - some models held steady at 95% accuracy and then nosedived to 60% once the input crossed a certain length.
Output limits are much smaller - even models with 1M input contexts typically cap output at 8K-65K tokens. The context window is asymmetric - you can feed the model a lot, but it will not write a novel-length response in one go.
The cost math nobody computes before they build
Here is a number worth sitting with. Filling a 1M-token window costs $0.14 on DeepSeek V4 Flash and $10.00 on Claude Fable 5 - a 71x spread.
At Claude Sonnet 4.6's published rate of $3.00 per million input tokens, a single 200K-token request - a longish document review - costs $0.60 in input tokens alone. Do that 1,000 times a month and it's $600, before output. Switch to 20 tight RAG retrievals of ~2K tokens each (40K total input per query) and the same 1,000 queries run $0.12 each, or $120 total - a 5x reduction, with better accuracy because you're feeding the model only the relevant chunks rather than the whole document.
That is the practical argument for retrieval-augmented generation: not that it lets you work around token limits, but that it exploits the shape of attention. Simply cramming more documents into a longer context doesn't guarantee better performance. Strategic placement of important information at the beginning or end of prompts often yields better results.
Summarizing aggressively helps too - a 500-token summary of a 10K document often outperforms stuffing the whole document in context.
Beagle in action#legal, 2:47pmWhat this means when you're building on top of a model
A few things follow directly from the mechanics above.
Order matters more than size. Models show strong recency bias (favoring information at the end) and primacy bias (favoring information at the beginning), but struggle with middle sections. If you have critical instructions, put them at the top. If you have a key fact the model needs to reason about, don't bury it between two large boilerplate sections.
Context rot starts before the limit. A model with a 200K token window can exhibit significant degradation at 50K tokens. The advertised number is a ceiling, not a comfort zone. A practical rule: plan around 40-50% of the advertised window as your effective size for tasks requiring precise retrieval and reasoning. For a 128K model, target 50-60K usable tokens. For a 1M model, target 400-500K. You can send more, but response quality will be measurably better with curated context than with maximum context.
Distractors compound the problem. It's not just length. Chroma's 2025 study found that semantically similar but irrelevant content actively misleads the model, causing degradation beyond what context length alone explains. A single distractor reduced baseline performance, and four distractors compounded the effect further. A dense context full of documents that are almost relevant is worse than a short context with only the right one.
Long-context models are improving but haven't closed the gap. Google's research found that Gemini 2.5 Flash can answer needle-in-a-haystack questions with great accuracy regardless of document position, suggesting substantial improvements in long-context retrieval. But simple factoid retrieval and complex multi-step reasoning across a long document are very different tasks - improvements on the former don't guarantee the latter.
A teammate like Beagle, working inside Slack, keeps context tight by design: each answer is drawn from the specific source documents that match the question, rather than everything in a channel's history. That's not a product constraint - it's what the attention research recommends.
LLM context windows: common questions
What is a context window in an LLM?
The context window is the total number of tokens - roughly ¾ of a word each - that a model can process in a single call. It holds everything at once: your prompt, system instructions, retrieved documents, conversation history, and the model's own response. Exceed it and old content is dropped or the call fails.
Why does adding more context make the model less accurate?
As context grows, the model's finite attention budget gets spread thinner across more tokens. Information that was highly attended at 1,000 tokens may be functionally ignored at 100,000 tokens. The lost-in-the-middle effect means content positioned in the center of a long context receives systematically less attention than content at the edges, regardless of relevance.
What is "lost in the middle" in LLMs?
Lost in the middle is a documented failure mode where models attend well to the start and end of a context but poorly to the center. Stanford and UC Berkeley first documented it: accuracy dropped by more than 30% when relevant information was placed in middle positions compared to positions near the beginning or end. The fix is to place critical information at the top of the prompt, or use retrieval to serve only the relevant chunks.
How does context window size affect API cost?
Tokens drive cost and latency: APIs bill per token, usually with separate input and output rates. A long prompt or a verbose response directly costs more, so trimming context is real money at scale. At current rates, a frontier model call with a 200K-token input costs around $0.60 in input alone - the same task done with 40K tokens of retrieved context costs roughly $0.12.
What is context rot?
Context rot is the performance degradation that happens when LLMs have to process increasingly long input contexts. The model's performance degrades when it has to search through longer contexts to find relevant information, even though that information is technically available in the window. It begins well before the token limit is reached - often past 50K tokens for a 200K-window model.