A single Llama 3 70B request at 128K context needs 42 GB of GPU memory just for the KV cache - leaving almost nothing for model weights on an 80 GB card, and zero room for concurrent users. That one number explains more about why AI products behave the way they do than any model card ever will.
The context window is the thing every AI user hits eventually. You paste in a long document, ask a question about page one, and the model gets it wrong - even though the text is right there. Or you're 40 messages into a conversation and the assistant contradicts something it said earlier. The standard explanation is "it ran out of context." That's technically true and practically useless. Here is what is actually happening.
How context windows actually work under the hood
The context window is the model's working memory: your prompt, system instructions, conversation history, tool outputs, and the model's own response all share the same token budget.
One token is roughly 0.75 English words, or about 4 characters.
A 128K token context window holds about 96,000 words - roughly 300 pages of text - or 50,000-70,000 lines of code.
The limit is not arbitrary. The self-attention mechanism computes representations by attending to all input token pairs simultaneously - and this imposes a fundamental quadratic cost: memory and compute scale as O(n²) in sequence length, creating a steep practical ceiling on context expansion. Double the context, and you quadruple the computation. That is the wall every model builder is climbing.
Three constraints create these limits: O(n²) complexity in self-attention, KV cache memory growth, and GPU memory bandwidth. The KV cache is worth understanding specifically. During inference, the model stores "key" and "value" representations of every token it has already processed, so it does not have to recompute them on every generation step. For a 7B-parameter model, 10,000 tokens stored in float16 precision require approximately 4.88 GB of KV cache memory. Scale that up: 125 GB of memory is necessary for storing the KV cache when the context length reaches 1 million tokens - well beyond the capacity of even high-end 80 GB GPUs.
Above 32K tokens, KV memory consumption starts outpacing parameter memory; above 128K it dominates; at 1 million tokens, KV cache eats 70-90% of available GPU VRAM. This is why bigger context is not free - it is directly and linearly expensive in hardware terms, even when token pricing might suggest otherwise.
The capacity vs. fidelity gap (the number that actually matters)
Here is the non-obvious part. The key distinction is between context capacity - how much a model accepts - and context fidelity - how well it uses what it accepts. These are not the same number, and the gap between them is large.
Models often perform worse on information buried in the middle of long contexts - the "lost in the middle" problem. Effective utilization is closer to 30 to 60 percent of stated window size for most current models.
The mechanism behind this is structural, not incidental. The architectural root cause lies in the RoPE long-term decay property: reduced dot-product similarity between distant token pairs systematically decreases attention weight on mid-context information. Softmax normalization amplifies this by concentrating attention on the highest-scoring tokens, reinforcing primacy and recency advantages.
The practical result: the "Lost in the Middle" paper from Stanford demonstrated that models perform significantly worse at retrieving information from the middle of long contexts compared to the beginning or end. Performance degrades in a U-shaped curve - the edges get attention, the middle gets forgotten.
This has been confirmed at scale. Independent evaluations of Llama 4 Scout indicate that its reliable reasoning degrades substantially beyond the 128K-256K range, even though the model physically accepts far more - while models tuned for long-context retention hold accuracy much better across their window. One research team building a long-running agent system caps the effective context window conservatively per provider - for example, 200K tokens for models that advertise 1M - informed by degradation studies and internal multi-needle retrieval tests.
The degradation is not always gradual. Grok 4 Fast shows strong degradation after 50K tokens even on benign tasks, deteriorating to near zero at 200K - despite a declared 2M token context window.
What this means for a real workday task
Say someone on your team asks an AI assistant to review a 40-page engineering spec and flag any contradictions with your existing architecture decisions. The spec alone is roughly 20,000 tokens. Add the system prompt, the architecture notes you pasted in, and a few prior conversation turns, and you're at 35,000-40,000 tokens before the model writes a single word of response.
That fits comfortably inside a 128K window - technically. But if the architecture decisions live in the middle of that 40,000-token block, the model may not reliably attend to them. When a fact is stated early in a long context and many tokens accumulate after it, the model's attention to that fact decreases. This is a measurable phenomenon tied to transformer architecture, not a hypothesis.
The right fix is not always "use a bigger model." RAG retrieves only the relevant document sections rather than stuffing everything into the context window
- and a shorter, well-ordered context often outperforms a bloated one. Place the most important evidence at the start and end of the prompt. Push weaker supporting passages toward the middle where attention is thinnest. Keep the core instruction outside the middle band under all conditions.
For agents running extended sessions, the problem compounds. Context rot matters more for agents than for simple chatbots because agents are doing more complex, multi-step work. A single-turn question-answer interaction rarely surfaces the problem. Agents that run extended sessions, coordinate with other agents, or maintain state across many tool calls are where context rot causes real damage.
Beagle in action#product-eng, 2:47pmHow the industry is closing the gap
In 2025 and 2026, several models crossed the 1-million-token barrier. Advances in efficient attention algorithms, better hardware, and techniques like ring attention made this possible - letting models process longer sequences without the cost scaling as sharply as it once did.
On the architecture side: sparse attention replaces full O(n²) attention with selective patterns - combinations of sliding windows for local context, global tokens that attend to everything, and random connections - reducing complexity from O(n²) to O(n) for fixed pattern sizes.
On the serving side: DeepSeek's Multi-head Latent Attention (MLA) is arguably the most important architectural innovation of the last two years. It compresses the KV cache and decompresses it on the fly, cutting memory usage by over 90% with no loss in quality.
But none of these eliminate the fidelity gap. Larger context windows help but do not solve the underlying attention degradation problem. The workarounds that actually work in production are RAG (retrieve rather than stuff), context summarization (compress old turns before they drift to the middle), and multi-agent architectures that give each agent a bounded, focused scope. Across 18 current models, all 18 degrade well below their maximum window length. Against the spatial problem, put less in the middle. Against the temporal one, start a fresh session earlier.
The context window is not a bug being fixed. It is a fundamental property of how these models compute. Understanding that - and designing around it rather than against it - is where the real leverage is.
How LLM context windows work: common questions
What is a context window in an LLM?
An LLM context window is the maximum number of tokens a model can process in one request. It is the model's working memory: your prompt, system instructions, conversation history, tool outputs, and the model's own response all share the same budget. When the total exceeds the limit, the model either fails or truncates earlier content.
Why do LLMs forget things in long conversations?
LLMs are stateless: they do not inherently remember past interactions. Each new request starts from zero. In a long conversation, earlier messages fill the context window, and the "lost in the middle" effect means the model attends less reliably to content in the center of the window even when it is technically present. The model has not forgotten - it is just not attending.
Why does a 1M token context window still fail on long tasks?
Capacity and fidelity are different things. Being able to accept 10 million tokens does not guarantee reasoning over them. A model may physically hold 1M tokens but reliably reason only over the first and last portions. Research teams building production agents routinely cap effective usage at 200K tokens even on models that advertise 1M, because degradation becomes measurable well before the limit is reached.
What is the KV cache and why does it matter?
The KV cache stores key and value representations of every already-processed token so the model does not recompute them during generation. It is what makes long-context inference possible at all - but it is also the primary GPU memory cost. KV cache now often exceeds model weights in memory consumption for long-context workloads. This is the main reason long-context API calls cost more per token than short ones.
What is "lost in the middle" and how do I avoid it?
Lost in the middle is a well-documented effect where retrieval accuracy follows a U-shaped curve across the context: information at the beginning and end is attended to reliably, while content in the center is deprioritized. The lost-in-the-middle effect can cut mid-context accuracy by more than 30%. To avoid it: put your most critical facts first or last, use RAG to retrieve only relevant sections rather than pasting full documents, and keep your prompts shorter than you think you need.