The original transformer, published in the 2017 "Attention Is All You Need" paper, had a context window of 512 tokens. That 512-token ceiling held for early transformer models through GPT-1. By April 2025, Meta's Llama 4 Scout shipped with a 10-million-token window - roughly 78 times larger than Llama 3's 128K limit , and equivalent to about 7.5 million words, or 15,000 pages. That jump of four orders of magnitude happened in under a decade, and most of the engineering required to pull it off is still invisible to the people who use these models every day. Here's what is actually going on.
What a context window actually is
The context window covers both input and output tokens combined. A model with a 128K context window can handle 128,000 tokens total - if your input uses 100K tokens, the model only has 28K tokens left for its response. That is the most practical thing to know, and most people miss it.
When you interact with an LLM, the context window includes system instructions, all previous messages in the current session, any retrieved documents you inject, and the model's own reply - all competing for the same fixed space. You burn through tokens fast, especially when system prompts, few-shot examples, and conversation history stack up.
In English, one token is roughly 0.75 words, so 1,000 tokens is approximately 750 words.
The ratio shifts dramatically with language: one October 2024 study found that a Telugu translation with far fewer characters than its English equivalent produced over seven times the number of tokens in context. If you are building anything that handles non-English text, this is not a footnote - it will hit your budget.
Why the attention mechanism makes long contexts expensive
Under the hood, a context window is tied to the model's attention mechanism. Transformer-based LLMs use self-attention to let every token "attend to" every other token in the sequence. The computational cost scales quadratically with sequence length - doubling the context window roughly quadruples the computation for the attention layers.
Put that concretely: using a 100K token context can require about 10,000 times more computation than a 1K context under a standard dense attention pattern. This is why the race to longer contexts is not just a matter of marketing - it required real architectural work.
The attention weight matrix is n×n, requiring O(n²) memory. For n=32K tokens at fp16, the attention matrix alone is approximately 2GB per head. That is before you add the KV cache.
The KV cache: the trade that makes inference possible. Every time the model generates a new token, it needs the key and value vectors for every previous token. Without caching, a transformer computes attention over all previous tokens when generating each new token. Without caching, generating 1,000 tokens requires recomputing attention from scratch 1,000 times. KV caching stores key and value tensors from previous tokens, reusing them for subsequent attention calculations - each new token computes attention against cached values rather than regenerating them.
The cost of that trade is memory. A 128K context window requires 32 times more cache memory than a 4K window. For a 70B model like Llama 3.1, at 128K context with a single request, the KV cache adds 40GB - and with just four concurrent requests at that context length, the KV cache alone would need 160GB, pushing totals beyond what a single H200 (141GB) can handle. This is why "we support 128K tokens" from a hosting provider comes with real infrastructure constraints that their pricing page does not always surface.
Model providers have invested heavily in architectural optimizations - sparse attention, ring attention, efficient KV-cache management - to make million-token contexts feasible, but the fundamental cost relationship remains.
How positional encoding determines what the model can "see"
Attention alone does not tell a model where in a sequence each token sits. Position encodings tell the model where each token sits in the sequence. Without them, attention treats "The cat sat on the mat" and "mat the on sat cat The" identically.
Most position encoding schemes fail catastrophically when asked to encode positions beyond their training range. This is why a model "trained at 256K" does not automatically generalize to 1M tokens without further work. For Llama 4 Scout, Meta used a technique called iRoPE - interleaved Rotary Position Embeddings. Instead of standard positional encodings, Scout uses interleaved attention layers without fixed positional embeddings, improving its ability to generalize to longer contexts, plus inference-time temperature scaling of attention weights to further enhance performance on long sequences.
Llama 4 Scout is both pre-trained and post-trained with a 256K context length, which empowers the base model with advanced length generalization capability
- the iRoPE architecture then lets it stretch to 10M tokens at inference time without a proportional training cost.
Beagle in action#product-ops, 2:47pmThe "lost in the middle" problem that big context windows don't fix
This is the non-obvious piece that context window size numbers obscure. Research from Stanford and the University of Washington demonstrates that LLMs exhibit a U-shaped performance curve when processing long contexts. Models achieve highest accuracy when relevant information appears at the beginning or end of the input context, but performance degrades significantly when critical information is positioned in the middle. This degradation occurs even in models explicitly designed for long-context processing.
Results showed that performance can degrade by more than 30% when relevant information shifts from the start or end positions to the middle of the context window.
This finding replicated across six model families - GPT-3.5-Turbo, GPT-4, Claude 1.3, LongChat-13B, MPT-30B, and Cohere Command.
The architectural root cause lies in the RoPE long-term decay property: reduced dot-product similarity between distant token pairs systematically decreases attention weight on mid-context information. Softmax normalization amplifies this by concentrating attention on the highest-scoring tokens, reinforcing primacy and recency advantages.
What does this mean practically? For many tasks, a context of 4-8 well-chosen chunks (roughly 2-4K tokens) outperforms a context of 20+ chunks (10K+ tokens). A million-token window does not fix a retrieval problem - it can make it worse if you fill it carelessly. Put your most important instructions at the top or bottom of the prompt. If you are building a RAG system, order your retrieved chunks so the highest-confidence results appear at the edges, not the middle.
A teammate like Beagle, running inside Slack or Teams, faces this constraint on every lookup it does. Keeping retrieved context tight - and sourced - is not just good hygiene; it is the difference between an answer that cites the right paragraph and one that summarizes the wrong section of a long doc.
How LLM context windows work: common questions
What is a context window in an LLM?
A context window is the total number of tokens - input plus output combined - that a model can process in a single request. Everything in the prompt (system instructions, conversation history, retrieved documents) and everything the model generates counts against the same fixed limit. Exceed it and either the API rejects the request or older content is silently dropped.
Why does a longer context window cost more?
The self-attention mechanism compares every token to every other token, so computation scales quadratically with sequence length. Doubling the context window quadruples the attention cost. Additionally, the KV cache (which stores key and value vectors for each past token) grows linearly with context length, consuming GPU memory that quickly exceeds even the model's own weights at long contexts.
What is the "lost in the middle" problem?
Models reliably recall information placed at the very beginning or very end of a long context, but performance degrades by 30% or more when critical information sits in the middle. This is an architectural effect tied to how attention weights decay with distance, and it persists even in models built specifically for long-context work. Keeping prompts short and placing key facts at the edges is the practical fix.
What is a KV cache and why does it matter?
The KV cache stores the key and value tensors computed for each token, so the model does not have to recompute them from scratch every time it generates a new token. Without it, generating 1,000 tokens would require 1,000 full forward passes. The tradeoff is GPU memory: a 128K context window requires 32 times the cache memory of a 4K window, which is why long-context inference often needs specialized infrastructure.
How did context windows get so large so fast?
The original 2017 transformer had a 512-token window. Three changes drove the expansion: efficient attention mechanisms (FlashAttention, sparse attention, grouped-query attention) that reduce the quadratic memory cost; improved positional encoding schemes (RoPE and its variants like iRoPE) that let models generalize to positions beyond their training length; and simple scale - training on more data with longer sequences. Llama 4 Scout's 10M-token window required all three, combined with interleaved attention layers that shed fixed positional embeddings entirely in some layers.