Understand the LLM Context Window Before You Trust It

The original 2017 transformer ran on 512 tokens. That first model used 512 tokens - and the limit was GPU memory, not architecture. Today's frontier models advertise 1 million tokens. That's a 2,000× leap in seven years. What hasn't changed: the model's relationship to where information sits inside that window is still deeply uneven. Understanding that gap is what separates people who use context windows well from people who paste in 80,000 tokens and wonder why the output is soft.

What a context window actually contains

A context window is the maximum number of tokens an LLM can process in a single request. It includes everything: your system prompt, user message, any documents or examples you include, and the model's generated response. Once you exceed it, the model cannot process the additional text.

Tokens aren't words. English prose tokenizes at roughly 1.3 tokens per word. Code is denser - closer to 1 token per 3.5 characters - and non-Latin scripts can be much denser still. So when a model advertises 200,000 tokens, the practical ceiling in plain English prose is around 150,000 words - roughly a thick technical manual. A million tokens gets you toward 750,000 words, enough for several textbooks or a medium-sized codebase.

The window's contents are fixed per request. There is no scroll. The model cannot reach back to something that fell off an earlier exchange unless that exchange is still inside the current window. This is why chat products manage a rolling buffer behind the scenes, silently trimming old turns so you don't hit the wall.

How attention makes the window work - and why it scales badly

Every token in the context gets compared to every other token. That's the self-attention mechanism. Transformer models use self-attention to process tokens in a context window. Self-attention calculates the relationships and dependencies between different tokens, even those far apart in the input sequence - this is what lets the model understand how words at the beginning of a paragraph relate to those at the end.

The mechanics are a set of learned weight matrices - queries (Q), keys (K), and values (V) - all derived from the input. The model projects each token into all three spaces, then computes a dot product between every Q and every K to produce an attention score, runs those scores through a softmax, and uses the result to weight the V vectors. The output at each token position is a weighted sum of the entire context.

The self-attention mechanism imposes a fundamental quadratic cost: memory and compute scale as O(n²) in sequence length n, creating a steep practical ceiling on context expansion. Double the context and you quadruple the attention computation. This is the central reason long context is expensive.

The cost shows up directly in GPU memory. Although a KV cache can reduce decoding latency, it demands a huge amount of GPU memory for long contexts. For Llama-3-8B, 125 GB of memory is necessary for storing the KV cache when the context length reaches 1 million tokens - far beyond the capacity of a RTX 4090 (24GB) or even an A100 (80GB).

As models support 128K and 1M token context windows, the KV cache can easily exceed the model weights in total memory usage. For Llama 3.1 70B at 128K context with a single request, the KV cache adds 40 GB - roughly 29% of the total.

The upshot: when a cloud provider advertises a million-token window, they're running your request across multiple GPUs, sharding the KV cache. You pay for that in latency and price per token - often with a hard multiplier. Some providers apply long context pricing multipliers when you exceed certain thresholds. GPT-5.4 charges 2× for input tokens beyond 272K. Gemini 3.1 Pro charges 2× beyond 200K tokens.

The problem no one mentions in the headline: lost in the middle

Here is the non-obvious part. A million-token window and a million-token effective window are different things.

Performance can degrade significantly when the position of relevant information changes. In particular, performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts - even for explicitly long-context models.

This was documented across six model families - GPT-3.5-Turbo, GPT-4, Claude, and others - and the pattern held in each. Accuracy is highest when relevant information appears at the beginning or end of the input context and degrades by more than 30% when relevant information is positioned in the middle.

The root cause is architectural. The architectural root cause lies in the RoPE long-term decay property: reduced dot-product similarity between distant token pairs systematically decreases attention weight on mid-context information. Softmax normalisation amplifies this by concentrating attention on the highest-scoring tokens, reinforcing primacy and recency advantages.

The RULER benchmark demonstrates that the effective context length of models is often far below their advertised maximum, with task-dependent degradation. A model that claims 1M tokens may produce reliable recall at 100K and start drifting long before the ceiling.

Current window sizes, costs, and where the market sits

The context race has produced numbers that were unthinkable three years ago.

Claude Opus 4.6, Claude Sonnet 4.6, Google's Gemini 3.1 Pro, Gemini 3 Flash, and Meta's Llama 4 Maverick sit at 1M tokens as the current frontier for production workloads. Claude Opus 4.6 and Sonnet 4.6 reached full GA of their 1M context window on March 13, 2026 - at standard pricing with no long-context surcharge. A 900K-token request costs the same per-token rate as a 9K one. That is a meaningful pricing distinction from OpenAI's approach.

A medium codebase of 10,000 lines of code is approximately 114,000 tokens, which fits in any model with a 128K+ context window. Larger codebases of 50,000+ lines may require models at the 1M token range.

The practical ceiling for most teams is still well under 128K per request. A 60-minute meeting transcript lands around 10,000 tokens. A 50-page report is roughly 25,000. You hit 128K when you start ingesting entire repos or multi-week conversation logs. Beyond that, you are paying the 2× multiplier or running a self-hosted stack with the memory to match.

One practical consequence of the lost-in-the-middle finding: for agentic AI workflows running long-horizon tasks, a 1M context window lets the entire trace stay intact - every tool call, observation, and intermediate reasoning step - which eliminates the compaction and context-clearing that used to cause agents to lose the plot mid-task. That is a genuine gain. But it only holds if the agent's reasoning architecture puts the most recent and most critical context at the boundaries of the window, not buried in the middle.

LLM context window: common questions

What is a context window in an LLM?

A context window is the total number of tokens - including your system prompt, documents, conversation history, and the model's output - that a model can process in one request. Once the window fills, earlier content is no longer visible to the model. Typical production models today support 128K to 1M tokens.

Why does the LLM context window have a memory cost?

Every token the model processes adds to a key-value (KV) cache that must sit in GPU memory during inference. The KV cache is the hidden memory driver in LLM inference - during autoregressive generation, the model stores key and value vectors for every previously processed token so it does not have to recompute them. Context length is the most overlooked driver of GPU memory consumption. At 1M tokens, that cache alone can exceed 125 GB on an 8B-parameter model.

Does a bigger context window mean better answers?

Not automatically. A larger context window means the model can accept more input, but quality, latency, price, retrieval accuracy, and provider-specific limits still matter. The "lost in the middle" effect means that cramming a window with marginally relevant content can dilute attention on the parts that matter most.

How should I structure prompts for long context windows?

Put the most important information first or last. The middle of a long context is where recall performance drops most. If you're feeding multiple documents, put the one with the key answer at position one or at the end - not buried at document seven of twelve. For structured retrieval tasks, keep only the three to five most relevant documents rather than retrieving twenty.

What is the difference between context window and model memory?

A context window is ephemeral - it resets between API requests unless you explicitly carry it forward. Model memory in the agent sense refers to external storage (databases, vector stores, conversation logs) that gets retrieved and injected into the context window per request. The window is the RAM; external memory is the disk. A teammate like Beagle maintains thread history as external context that gets surfaced into the window only when it's needed, keeping the window lean.