Three hours into a coding session with Claude, a developer noticed the generated code had quietly drifted. Variables were renamed. Architectural patterns had changed. The model wasn't broken - it had silently dropped the original instructions from its context window. Three hours of conversation had pushed those early messages out. The bill for that session included several dollars of API cost to re-explain things the model had already processed once.
This is the most misunderstood thing about modern AI models. The context window isn't a reading comprehension limit. It's a shared workspace with fixed dimensions, a U-shaped blind spot, and a memory bill that compounds every turn.
What the context window actually is
The context window is the total number of tokens an LLM can process in a single request - your prompt, conversation history, uploaded documents, system instructions, and the model's response all compete for the same fixed space. Think of it less like human short-term memory and more like a whiteboard: once it's full, something gets erased to make room.
As a rough approximation, one token represents about four characters or three-quarters of a word , so a 128K-token window holds something like 96,000 words - roughly a standard novel. That sounds enormous until you factor in system prompts, retrieved documents, tool outputs, and the model's own responses, which all draw from the same budget.
Context windows have expanded by roughly two orders of magnitude since the original transformer architecture, from a few thousand tokens in early models to 1-2 million tokens today.
As of June 2026, thirteen hosted frontier models ship 1M+ token windows: Claude Fable 5, Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4, MiniMax M3, and Qwen3.5-Plus, among others.
But the advertised number hides the part that matters operationally.
How attention actually works - and where it fails
Every modern LLM is built on transformers. The core operation is attention: for each token the model generates, it looks back at every previous token and decides how much to "attend" to each one.
In plain terms: the model converts input into query (Q), key (K), and value (V) vectors, computes similarity scores between queries and keys, normalizes them, and uses those scores to weight the values.
The key constraint: attention is O(n²) in sequence length. Double your context, quadruple the compute. This is why longer contexts cost more - it's not linear, it's exponential, at least on naive implementations.
Here is the part that rarely appears in product marketing. The "lost in the middle" effect is a demonstrated phenomenon where LLMs perform significantly worse when relevant information is placed in the middle of the input context rather than at the beginning or end.
Liu et al. (2024) showed performance drops of over 30% on multi-document QA when the answer document was at position 10 out of 20, compared to position 1 or 20.
The architectural cause is specific. Rotary Position Embedding (RoPE), used in most modern LLMs, introduces a long-term decay effect that makes models attend more strongly to tokens at the beginning and end of sequences while de-emphasizing middle content.
Softmax normalization amplifies this by concentrating attention on the highest-scoring tokens, reinforcing primacy and recency advantages.
The NoLiMa benchmark from LMU Munich and Adobe Research, presented at ICML 2025, found that when models couldn't rely on surface-level keyword matching, 11 out of 13 LLMs dropped below 50% of their baseline scores at just 32K tokens. GPT-4o fell from a near-perfect 99.3% baseline to 69.7%.
The implication is direct: if you're using RAG and stuffing 20 retrieved chunks into a prompt, the most critical chunk needs to land at the top or the bottom - not in the middle, where attention is thinnest.
The KV cache: why long sessions get expensive
There's a performance mechanism under the hood called the KV cache, and understanding it explains both the speed of modern AI and the economics of long conversations.
Transformer attention works by computing relationships between every token in the sequence and every other token that came before it. For each token, the model produces three vectors: a Query (Q), a Key (K), and a Value (V). The attention mechanism uses Q to find which K vectors are relevant, then aggregates the corresponding V vectors to produce the output.
The problem is the K and V vectors. For token 5,000 in a sequence, the model still needs the K and V vectors from tokens 1 through 4,999 to compute attention correctly. Without caching, the model would recompute all of this from scratch for every new token generated.
A KV cache stores intermediate key (K) and value (V) computations for reuse during inference, which results in a substantial speed-up when generating text. Instead of re-deriving attention relationships for your entire conversation history on every turn, the model loads pre-computed values from memory.
The tradeoff is RAM. At 32,768 tokens, that's roughly 80 GB of KV cache alone, which fills the entire VRAM of a single NVIDIA H100 SXM before model weights or activations enter the picture. At 128K tokens, you're looking at over 300 GB just for the cache. Modern architectures like grouped-query attention bring this down significantly, but it's why frontier model inference is expensive at scale.
For the person building in Slack or Teams: this is the reason a tool like Beagle, which holds conversation context across turns, has to be deliberate about what stays in the active window. The context window includes the entire conversation history that must be resent each turn to maintain coherence - every turn, the full conversation is re-sent to the model. Every message you add slightly increases the cost of every future message in that thread.
What to do with this in practice
Knowing the mechanics, four things follow directly.
Put critical text at the edges. Your system prompt and the most important retrieved context belong at the beginning. If you're building a RAG pipeline that ranks 10 retrieved chunks, don't put the best match at position 5. Put the critical stuff at the start or end, because models genuinely lose the middle.
Don't confuse window size with reliable attention. Most models market a context window range, but effective context often falls far below the advertised maximum.
A model claiming 200K tokens typically becomes unreliable around 130K, with sudden performance drops rather than gradual degradation. Test your actual workload, not the spec sheet.
Use RAG to keep the window clean. The key discipline is treating the context window as a high-value workspace rather than a storage bin. Only the most relevant tokens should be allowed in at any moment. This approach scales better, reduces cost, and often improves accuracy compared to dumping entire documents into the prompt.
Cache the static parts. If you send the same system prompt and knowledge base on every request, prompt caching recovers most of that cost. Claude charges $5.00 per million input tokens at standard rates but $0.50 per million for cached reads - a 90% discount on the part of your prompt that never changes.
The context window number in the headline is real. What the headline doesn't mention is the U-shaped curve, the KV cache bill, and the fact that your most important sentence probably shouldn't be in the middle of a long prompt. Those details are the difference between a model that works and one that almost works.