Paste a 100-page contract into Claude and something invisible is already constraining what happens next. The model doesn't read it the way you do - front to back, weighting every paragraph equally. It loads the whole document into something closer to working RAM, processes it in one shot, and research shows the paragraphs in the middle are the ones most likely to be missed.
That constraint has a name: the context window. Understanding how it works changes how you build with AI, what you pay, and why outputs degrade in ways that are hard to debug.
What a context window actually is
An LLM context window is the maximum amount of text - measured in tokens - that a model can process in a single request. Think of it as the model's working memory: everything you send in your prompt, any retrieved documents, conversation history, and the response all need to fit within this limit.
The token unit is easy to underestimate. As a rough rule of thumb in English, one token is about four characters, and 1,000 tokens is roughly 750 words - so a model with a 1 million token context window can hold around 750,000 words, or about 1,500 pages of text, in view at one time. Code is denser: code has more tokens per line due to special characters. A 1,000-line Python file might be 5,000-8,000 tokens.
What goes in the window matters as much as the size. When you interact with an LLM, the context window includes system instructions that guide the model's behavior (typically 100-1,000 tokens), all previous messages in the current session, and more. Every piece competes for the same fixed space.
The window is also asymmetric in a way most people don't notice. Output limits are much smaller. Even models with 1M input contexts typically cap output at 8K-65K tokens. The context window is asymmetric - you can feed the model a lot, but it will not write a novel-length response in one go. That matters if you're trying to generate a long document from a long source.
Why long contexts are expensive: the quadratic problem
Context window limits come from how transformers work. Three constraints create these limits: O(n²) complexity in self-attention, KV cache memory growth, and GPU memory bandwidth. Transformers need to compare every token to every other token.
The math is unforgiving. Double your context and you quadruple the work. A 10K token context needs 100 million comparisons. A 100K token context needs 10 billion. This is why your inference crawls with long contexts.
To avoid recomputing all that on every token generated, models use a KV cache - a structure that stores the attention states of tokens already processed. But the memory demand grows fast. Storing a KV cache for 100K tokens in a 7B-parameter model demands over 50GB of memory, whereas a 2K token context requires less than 1GB. That's a 50x memory increase for a 50x context increase - and it's why cloud providers charge more per token as contexts grow longer.
The "lost in the middle" problem
Here is the part that bites teams who assume filling the window is the same as the model reading the window.
Research from Stanford and the University of Washington demonstrates that LLMs exhibit a U-shaped performance curve when processing long contexts. Models achieve highest accuracy when relevant information appears at the beginning or end of the input context, but performance degrades significantly when critical information is positioned in the middle.
The degradation is measurable and large. With 20 retrieved documents totaling around 4,000 tokens, accuracy declined from 70-75% for information at positions 1 or 20 down to 55-60% when positioned in the middle. That's a 15-20 percentage point drop based entirely on position, not content quality.
The root cause lies in the attention mechanisms and positional encodings used by transformer-based models. Rotary Position Embedding (RoPE), commonly used in modern LLMs, introduces a long-term decay effect that causes models to prioritize tokens at the beginning and end of sequences while de-emphasizing middle content.
This isn't fixed by switching to a bigger model. Newer models have improved long-context capacity, but they still do not use every position equally. Research on context rot and effective context windows shows that performance can degrade before the advertised token limit.
The practical consequence: if you're building a system that stuffs ten retrieved documents into a prompt and expects the model to pull the right fact from document six, you are building on shaky ground. Put the most important context at the top or the bottom. Treat the middle as lossy.
How context window size has changed - and what it costs now
In early 2023, most models operated with 4K-8K token windows. By the end of 2025, leading models routinely support 200K tokens or more, with some reaching 1 million tokens or beyond. That's a roughly 250x expansion in under three years.
The current landscape looks like this:
| Model | Input context | Output limit | Note |
|---|---|---|---|
| GPT-4.1 | 1M tokens | ~32K tokens | Flat pricing, no surcharge at length |
| Gemini 2.5 Pro | 1M tokens | 65K tokens | Strongest for multimodal long-context |
| Claude Opus 4.6 | 200K tokens | varies | Best-rated for agentic accuracy |
| GPT-4o | 128K tokens | 16K tokens | Widely deployed baseline |
| DeepSeek V3 | 128K tokens | ~8K tokens | Open-weight frontier |
Some providers apply long-context pricing multipliers when you exceed certain thresholds - GPT-5.4 charges 2x for input tokens beyond 272K. A million-token request isn't just slower; it can be meaningfully more expensive than running the same content through a smarter chunking strategy.
Beagle in action#legal-ops, 11:02amHow LLM context windows work: common questions
What is a context window in simple terms?
A context window is the maximum amount of text an LLM can process in one request - your instructions, any documents you provide, the conversation history, and the model's reply all have to fit inside it. Think of it as working RAM: once it fills, older content either gets dropped or the request is rejected.
Why does a larger context window cost more?
Because attention computation scales quadratically with token count. Doubling the context roughly quadruples the attention work, and every generated token must read the full KV cache - a memory structure that grows linearly with context length. At 100K tokens, that cache can exceed 50GB for a 7B model, requiring expensive GPU memory bandwidth on every step.
Does a bigger context window mean better answers?
Not automatically. Research shows LLMs recall information at the beginning and end of context far more reliably than content in the middle - a well-documented effect called "lost in the middle." Performance on retrieval tasks can drop more than 30% for information buried in the center of a long prompt, regardless of window size.
What is context rot?
Context rot is the practical term for quality degradation that happens as context grows. Even when a model physically accepts 1M tokens, reasoning accuracy often degrades well before that limit - sometimes as early as 128K-256K tokens depending on the task. The spec-sheet limit and the reliable working limit are not the same number.
What fits in a 128K context window?
About 96,000 words - roughly the length of a full novel. In code terms, a medium codebase of around 10,000 lines fits comfortably. A 50-page dense technical document (roughly 100K tokens) will just fit, but leaves little headroom for conversation history and system instructions competing for the same space.