The original transformer, published in 2017, had a context window of 512 tokens - about one printed page. The limit was GPU memory, not architecture. Today, context window sizes range from 128,000 tokens for mainstream models to 10 million for Meta's Llama 4 Scout, and the 1-million-token window has become the new standard for frontier closed-source models. The spec sheet has never looked better. The performance has not kept up.
There is a gap between what a model accepts and what it uses well, and that gap has real consequences for anyone building on top of these systems. This post explains how a context window actually works at a mechanical level, what the "lost in the middle" problem is and why it persists, and what the numbers look like when you start filling these windows in production.
How the context window actually works, mechanically
The context window is the maximum number of tokens a model can consider at once - its entire working memory for a request. Everything you send plus everything the model generates must fit in the context window; exceed it and earlier content is truncated.
Tokens are not words. Large language models don't process text as words or letters - they process tokens, chunks of text produced by a tokenizer that sits between your text and the model. A token is often a whole common word, but longer or rarer words get split into pieces, and spaces and punctuation count too. As a rough conversion, one token is approximately 0.75 English words, so a 200,000-token window holds roughly 150,000 words, or about 500 printed pages.
The mechanism that reads all those tokens is called self-attention. LLMs use a self-attention mechanism that calculates the relationship between every token and every other token in the context. That "every token against every other token" part is the expensive part. Each token in the input has to be compared against every other token through a mechanism called self-attention, and the computational cost of this attention mechanism grows quadratically with the number of tokens - doubling your input length can roughly quadruple the compute required.
This is why context windows have a hard ceiling, and why moving that ceiling upward is an engineering problem, not just a configuration flag. A model with a 1,000,000-token window may still generate at most 128,000 tokens per turn
- the input and output budgets are separate caps, and models don't always advertise both clearly.
The "lost in the middle" problem is real and it hasn't gone away
Here is the part the marketing copy skips. Having a 1M-token context window does not mean the model reads all of it equally well.
Research from Stanford and the University of Washington demonstrates that LLMs exhibit a U-shaped performance curve when processing long contexts. Models achieve highest accuracy when relevant information appears at the beginning or end of the input context, but performance degrades significantly when critical information is positioned in the middle - and this degradation occurs even in models explicitly designed for long-context processing.
The numbers are specific. Accuracy degrades by more than 30% when relevant information is positioned in the middle, a finding that replicated across six model families including GPT-3.5-Turbo, GPT-4, and Claude. The underlying cause is architectural: the root cause lies in a long-term decay property in the positional encoding scheme, where reduced similarity between distant token pairs systematically decreases attention weight on mid-context information, and softmax normalisation amplifies this by concentrating attention on the highest-scoring tokens, reinforcing primacy and recency advantages.
In plain terms: the model behaves something like a person who reads the first paragraph of a document closely, skims the middle, and reads the last paragraph closely again. Newer models have improved, but newer models have improved long-context capacity, but they still do not use every position equally, and research on effective context windows shows that performance can degrade before the advertised token limit.
There is also a separate problem between capacity and fidelity. Independent evaluations of Llama 4 Scout indicate that its reliable reasoning degrades substantially beyond the 128K-256K range even though it physically accepts far more. The lesson is to distinguish context capacity (how much a model accepts) from context fidelity (how well it uses what it accepts).
Beagle in action#engineering, a long incident threadWhat filling a large context window actually costs
Filling a one-million-token window once costs between $0.30 and $10.00 at 2026 list prices, depending on the model tier. That is cheap. Filling it fifty times in one agent session is not, and that is the cost nobody quotes.
The spread across models is significant. Filling the same 1M window costs $0.14 on DeepSeek V4 Flash and $10.00 on Claude Fable 5 - a 71x spread for the same window size. Some models also charge a long-context surcharge on top. GPT-5.4 supports up to 1M tokens of input context via the API, but anything beyond the standard 272K token window triggers a pricing surcharge where input cost doubles from $2.50 to $5.00 per million tokens.
Here is how current frontier models compare on the window-to-cost trade-off:
| Model | Context window | Cost to fill 1M tokens | Notes |
|---|---|---|---|
| DeepSeek V4 Flash | 1M | $0.14 | Cache hits drop to ~$0.003/M |
| GPT-5.5 | 1M | $5.00 | 2× surcharge above 272K |
| Gemini 3.1 Pro | 1M | $4.00 | Surcharge above 200K |
| Claude Fable 5 | 1M | $10.00 | Highest quality, highest price |
| Llama 4 Scout | 10M | Self-hosted | Degraded recall beyond ~1M |
Prices from provider APIs as of June-July 2026. Verify before building.
When an AI agent executes a multi-step task - querying databases, calling APIs, reasoning over intermediate results - each step adds to the context. A 50-step workflow where each LLM call consumes 20,000 tokens means 1,000,000 tokens of total context over the course of the task. If the agent needs to reference early results in later steps, context accumulates within the window, not just across calls. That is the cost model nobody puts in the demo.
Why you still need retrieval even with a 1M-token window
The instinct when context windows grew to 1M tokens was to ask whether RAG was dead. It is not.
Larger context windows cost more to use, can introduce subtle attention degradation across very long inputs, and rarely outperform a well-designed retrieval pipeline for finding the right information. Most production systems use retrieval to surface the few thousand tokens of context that actually matter rather than stuffing everything into a giant window.
Because context is finite and costly, you don't dump everything into the prompt - instead you use retrieval-augmented generation to fetch only the relevant chunks, summarize long histories, and prune aggressively. A good retrieval pipeline puts the right information at the beginning or end of the context (avoiding the middle-position penalty), uses only what's relevant, and costs a fraction of filling the full window.
There is also a meaningful distinction between retrieval tasks and reasoning tasks at long context. The exception is tasks where the model only needs to find a specific piece of information (needle-in-a-haystack), where frontier models now perform well across the full window. Synthesizing hundreds of pages of mid-context material is a different, harder problem. Asking a model to find a specific contract clause in a 500-page document is one thing; asking it to reason across all 500 pages simultaneously is another.
A teammate like Beagle treats context as a budget: when a question lands in a Slack thread, it fetches the relevant Notion doc or policy, not the entire knowledge base, and places the most important information where attention is strongest.
LLM context windows: common questions
What is an LLM context window?
A context window is the amount of information an AI model can consider in a single request. It's measured in tokens - the basic units of text a language model reads and produces - and it sets a hard upper limit on how much input prompt, conversation history, and retrieved content the model can take into account when generating a response.
How many tokens is 1 million tokens in words or pages?
One token is approximately 0.75 English words, so a 200,000-token window holds roughly 150,000 words, or about 500 printed pages. Scaling that up, 1 million tokens is roughly 750,000 words - about 2,500 pages, or several long novels. Most real work uses a fraction of that; very few queries need more than 20,000 tokens of curated context.
What is the "lost in the middle" problem in LLMs?
It refers to the documented U-shaped accuracy curve in long-context models. Large language models struggle to effectively use information located in the middle of long contexts. This phenomenon, known as the "lost in the middle" problem, poses significant challenges for RAG systems that depend on accurately retrieving and utilizing relevant information from extended document collections. The fix: surface fewer, higher-relevance chunks and place the most critical facts near the start or end of the context.
Does a bigger context window mean better results?
Not automatically. Bigger windows do not automatically produce better results, and filling a 1M-token window costs anywhere from $0.09 to $30.00 depending on the model. Effective context - the portion a model actually uses reliably - tops out well below the advertised number. Better retrieval, not a bigger window, is usually the higher-leverage improvement.
How fast have context windows grown?
Anthropic's Claude reached 100,000 tokens in May 2023; Claude 2.1 reached 200,000 in November 2023; Google's Gemini 1.5 Pro reached 1,000,000 tokens in February 2024; and Meta's Llama 4 Scout reached 10,000,000 in April 2025.
Since mid-2023, the longest LLM context windows have grown by about 30x per year. The pace of growth has outrun the pace of quality improvement at long contexts - which is why understanding the limits still matters even as the numbers keep climbing.