How the LLM Context Window Actually Works

Paste a 30-page contract into Claude and it reads the whole thing without blinking. Paste the same contract plus three more like it and something subtle starts to break - not an error, just quietly worse answers from the middle pages. That's not a bug. It's the physics of how attention works, and it has direct consequences for every team building on top of a language model.

What the context window actually contains

The context window is the maximum number of tokens a model can consider at once - its entire working memory for a request. Everything you send plus everything the model generates must fit in it; exceed it and earlier content is truncated.

That "everything" is worth unpacking. The context window must hold everything at once: system instructions, conversation history, any pasted documents, the question, and the generated response. When your AI assistant in Slack has a long thread going, every prior message is re-sent to the model on each new turn - nothing is stored between calls. The window fills up from the left, and when it's full, the oldest material drops off.

Before text enters the context window, it converts into tokens through a process called tokenization.

LLMs don't read words or characters - they read tokens, chunks of text roughly 3-4 characters long. A useful rule of thumb: about 100 tokens per 75 English words, so 1,000 words is roughly 1,300 tokens. That 30-page contract? Probably 15,000-20,000 tokens before the model writes a single word back.

Why attention cost scales quadratically

Under the hood, the context window is tied to the model's attention mechanism. Transformer-based LLMs use self-attention to let every token "attend to" every other token in the sequence. The computational cost scales quadratically with sequence length - doubling the context window roughly quadruples the computation for the attention layers. This is why larger context windows are expensive.

Put numbers on it: a 10K token context needs 100 million comparisons; a 100K token context needs 10 billion. This is why your inference crawls with long contexts.

Model providers have worked hard to make this manageable. Techniques like Flash Attention and sparse attention chip away at it, but the fundamental relationship holds. Providers have invested heavily in architectural optimizations - sparse attention, ring attention, efficient KV-cache management - to make million-token contexts feasible, but the fundamental cost relationship remains.

The KV cache: what actually lives in GPU memory

Here is the part most explanations skip. The context window is not RAM in the ordinary sense - what actually sits in GPU memory is a structure called the KV cache.

A decoder-only transformer generates one token at a time. To produce the next token, the attention mechanism needs the key and value vectors for every token that came before it. The naive approach would recompute those keys and values for the entire sequence on every single step - quadratic work that gets slower as the text grows. The KV cache is the optimization that avoids it. When the model processes a token, it computes that token's key and value vectors once and stores them in GPU memory. Every subsequent step reuses them. Generation becomes linear in the number of new tokens instead of quadratic in the sequence length.

The catch: this cache grows with every token in the context window and with every concurrent request being served.

A 128K window on Llama 3 70B eats roughly 40 GB of KV cache for a single session. Stack concurrent users and KV cache routinely exceeds the model weights themselves as the dominant memory cost.

Context windows have grown from 1K-2K tokens in early GPT-style models to 4K in Llama 2, 128K in Llama 3.1, and longer in recent commercial systems; because KV memory scales linearly, a model needing 1.25 GiB of KV at 4K reaches 40 GiB at 128K and exceeds 300 GiB per request at 1M context.

This is why a hosted API and a self-hosted model have completely different failure modes. On a hosted API, you hit a token limit and get an error. On hardware you own, you hit a VRAM wall - often without a clean error, just slowdown and then an OOM crash.

The "lost in the middle" problem: capacity vs. fidelity

Here is the finding that should change how you think about big context windows. Research demonstrated that LLM performance on multi-document question answering follows a U-shaped function of information position: accuracy is highest when relevant information appears at the beginning or end of the input context, and degrades by more than 30% when relevant information is positioned in the middle. This finding replicated across six model families including GPT-3.5-Turbo, GPT-4, Claude 1.3, and others.

The architectural root cause lies in the RoPE long-term decay property: reduced dot-product similarity between distant token pairs systematically decreases attention weight on mid-context information. Softmax normalisation amplifies this by concentrating attention on the highest-scoring tokens, reinforcing primacy and recency advantages.

Newer models have improved, but the effect persists. Newer models have improved long-context capacity, but they still do not use every position equally. Research on context rot shows that performance can degrade before the advertised token limit.

This creates a distinction worth keeping in your mental model: context capacity (how much a model accepts) versus context fidelity (how well it uses what it accepts).

Research showed that even when a model retrieves the right evidence, the sheer volume of surrounding distractor text degrades its ability to apply that evidence. For tasks that require deep logic across a large corpus, a well-built RAG system that feeds the model only the relevant chunks often beats stuffing everything into a giant context window.

The practical implication: if you're building a system that drops a large document into context and asks questions about section 7, put section 7 near the top or the bottom of what you send. Do not bury it in the middle between the system prompt and a long conversation history.

A concrete example from first principles

Imagine a support engineer using an AI assistant to answer a question about a refund policy. The request goes to the model as one flat blob:

1. System prompt - "You are a support agent for Acme Co…" (~200 tokens)
2. Retrieved policy doc - the refund section pulled from a knowledge base (~800 tokens)
3. Conversation history - the last five turns of the chat (~600 tokens)
4. User's new message - "Can I get a refund after 45 days?" (~15 tokens)

Total: roughly 1,615 tokens. A 128K model processes this trivially. The KV cache holds ~500 KB for this session (negligible). The model can see everything.

Now scale that to an enterprise deployment with 300 concurrent support agents, each with a 10K-token average session. At Llama 3 70B's ~0.31 MB per token, the KV cache across the fleet runs to roughly 930 GB - more than ten H100 GPUs just for cache storage. That is the number most teams do not compute before they pick a context window size.

A teammate like Beagle, handling lookup questions inside Slack or Teams, keeps sessions short by design: the question arrives, the relevant doc gets retrieved (not the entire knowledge base), and the answer goes out. The context stays narrow, the KV footprint stays small, and the model's attention stays focused on what matters.

LLM context window: common questions

What is an LLM context window?

The context window is the total number of tokens a model can process in one request - inputs plus outputs combined. It is the model's entire working memory for that request. Anything outside the window is invisible to the model unless summarized or retrieved again.

Why does the context window affect speed and cost?

APIs bill per token, and attention cost grows with sequence length, so long prompts are slower and pricier. Prefill time - the time before the first token of the reply appears - grows roughly quadratically with context length. A 100K-token prompt takes orders of magnitude longer to prefill than a 1K-token one, even before billing.

What is the "lost in the middle" problem?

It is the finding that LLMs consistently recall information placed at the beginning and end of a long context better than information placed in the middle. Performance can degrade by more than 30% when relevant information shifts from the start or end positions to the middle of the context window. Putting your most important content near the edges of what you send is the simplest mitigation.

Does a bigger context window mean better performance?

Not automatically. A larger context window allows the model to handle longer prompts, but more context isn't automatically better. As token count grows, accuracy and recall degrade - a phenomenon known as context rot. This makes curating what's in context just as important as how much space is available.

How do I reduce KV cache memory at scale?

Prefer GQA/MQA models for any local long-context work - it's the difference between 40 GB and 320 GB at 128K. Turn on FP8/INT8 KV quantization before reaching for a bigger GPU; it's a free halving of the most expensive term. On hosted APIs, keep sessions short and retrieve only the relevant chunks rather than passing full documents.