Vendors have been selling context length the way they once sold clock speed: bigger is better, full stop. Gemini 2.5 Pro ships with a 2M-token window. Several models now advertise 10M. The implicit pitch is that if you can fit your entire history into the prompt, you do not need a memory system at all. A benchmark paper accepted at ICLR 2026 ran the experiment properly, and the result is uncomfortable for that pitch.
What BEAM actually measures - and why it differs from NIAH
Long-context benchmarks and memory benchmarks are not the same thing, and conflating them is the source of most vendor confusion. A long-context benchmark gives the model a single large fixed input and asks it to retrieve, summarize, or reason over the whole thing in one pass. Nothing was written. Nothing persists after the call. The Needle-in-a-Haystack test is the canonical example: one planted fact, one retrieval, done. NIAH inserts one fact into a long passage and asks the model to retrieve it - by 2026 it is a smoke test for context-window functionality, nothing more.
BEAM asks a harder question. It scales evaluations up to 10 million tokens across 100 procedurally generated, coherent multi-turn conversations, and tests 10 distinct memory dimensions including abstention, contradiction resolution, event ordering, instruction following across time, and preference tracking.
Many older benchmarks construct "long" conversations by stitching together sessions from different users, which creates abrupt topic shifts that ironically make the task easier - models can rely on local retrieval instead of true long-term memory. BEAM uses coherent single-user narratives to close that loophole.
The benchmark comes from the paper "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs," accepted at ICLR 2026.
The numbers: what happens as context scales to 10M tokens
The headline result is not subtle. Across all conversation lengths - from 100K to 10M tokens - BEAM consistently shows that structured memory systems outperform both standard long-context LLMs and RAG baselines. At shorter contexts, the gains are already significant, with improvements of over 40-50% on models like GPT-4.1-nano and Llama variants.
Then it gets worse for the "just make the window bigger" argument. As context grows, the real story appears. At 1M tokens, improvements from structured memory climb as high as 75%. At 10M tokens - where most models cannot even process full context - gains exceed 100% in some cases.
The BEAM paper also released LIGHT, a cognitive-science-inspired memory framework, alongside the benchmark. The headline finding is a persistent gap between long context alone and long context plus structured memory. LIGHT reports 3.5% to 12.7% higher accuracy on probing questions versus the strongest long-context baselines, with the gap widening as token scale grows. Models tested include GPT-4.1-nano, Gemini-2.0-Flash, Qwen2.5-32B, and Llama-4-Maverick.
Here is the raw comparison across four evaluated frameworks at two scales, pulled from the BEAM paper:
| System | 100K tokens | 10M tokens | Drop |
|---|---|---|---|
| RAG baseline | 32.3% | 24.9% | −23% |
| LIGHT (paper's own framework) | 35.8% | 26.6% | −26% |
| Honcho | 63.0% | 40.6% | −36% |
| Hindsight | 73.4% | 64.1% | −13% |
The structural divergence captures which architectures degrade gracefully under scale versus which fall off a cliff. Hindsight's lower degradation rate is worth noting: its 1M score (73.9%) is actually higher than its 500K score (71.1%), meaning performance does not degrade as token volume increases - it improves. Most systems show the opposite.
The BEAM 1M to BEAM 10M drop for the LIGHT framework (64.1 → 48.6) represents a ~25% performance loss as context scales 10x. That is the number to hold in your head when a vendor cites a 10M-token window as a reason to skip building memory.
Why context rot sets in before you hit the window limit
There is a second problem that does not show up in token-count headlines. Stuffing context does not actually work well even within a model's supported window. Research on context rot shows that LLM performance degrades significantly as input length increases. At 128K tokens, models are already losing coherence and missing information buried in the middle.
A 1M-token context window does not give you 1M tokens of reliable attention - it gives you diminishing returns most of the way there. This is why the BEAM result matters beyond the extreme 10M case: even at 128K, a structured memory system that retrieves the right 2,000 tokens beats a raw context dump at 10x the length.
For agent builders, BEAM is the benchmark that asks whether a 10M context window lets a team skip building memory. The paper's answer is no.
The practical architecture that follows from this: store everything, retrieve selectively. An agent operating at that volume has no choice but to retrieve selectively, and the quality of that retrieval is what separates functional memory from broken memory. The LIGHT framework operationalizes this with three components: long-term episodic memory, a short-term working buffer, and a scratchpad that distills salient facts as the conversation grows. At longer contexts, the scratchpad's contribution grows more pronounced - the gradual abstraction of dialogue content into compressed semantic notes is what allows LIGHT to scale to 10M tokens without exhausting the inference-time context budget.
Beagle in action#product, mid-sprintWhat multi-scope memory actually looks like in a production agent
The BEAM paper establishes the benchmark. The question for teams building agents is how to architect memory that does not collapse at scale.
Multi-scope memory is a design pattern where each memory write is tagged with identity scopes - user_id for facts that persist across sessions, agent_id for facts tied to a specific agent instance, run_id or session_id for conversation-scoped facts, and app_id or org_id for shared organizational context. These scopes are composed at retrieval time, and the pipeline merges and ranks results automatically.
That scope model matters for teams using agents in shared workspaces like Slack or Teams. An agent answering a question in #product should pull from org-level context (past decisions, docs, policies) and the current session - but not bleed another user's private conversation into the response. The scoping layer is what makes that separation explicit rather than relying on prompt instructions that are easy to override.
One more thing the benchmarks do not make obvious: evaluation cost. The evaluation cost for full BEAM reproduction is non-trivial - at 10M tokens per conversation, across 100 conversations, multiple backbones, and multiple memory configurations, full reproduction is a five-figure-USD inference budget. That is a research cost, not a production cost. A well-designed memory system avoids paying it on every request by only fetching what is needed.
A teammate like Beagle, operating inside Slack, reads from a scoped memory store rather than re-ingesting channel history on each request - the memory architecture page covers how that works in practice.
Does a bigger context window replace agent memory: common questions
What is the BEAM benchmark?
BEAM (Beyond a Million Tokens) is a memory benchmark accepted at ICLR 2026 that evaluates AI agents at 1M and 10M token scales across 10 memory dimensions including preference tracking, contradiction resolution, and temporal reasoning. Unlike NIAH or RULER, it uses coherent multi-user conversations to test true multi-session continuity, not single-pass retrieval.
Does a larger context window mean better memory?
No. BEAM shows structured memory systems outperform raw long-context models by 40-50% at short contexts and over 100% at 10M tokens. Context length measures how much an agent can read in one pass; memory measures whether facts persist and can be retrieved correctly across sessions. They are different problems.
What is context rot in LLMs?
Context rot refers to the well-documented degradation in LLM attention quality as input length grows - even within a model's supported context window. Research shows that at 128K tokens, models already lose coherence on facts buried in the middle of the context. A 1M-token window does not give you 1M tokens of reliable attention.
What is the LIGHT framework?
LIGHT is a memory-augmented framework released alongside the BEAM benchmark. It uses a cognitive-science-inspired three-layer architecture: long-term episodic memory, a short-term working buffer, and a scratchpad that compresses salient facts as conversation length grows. LIGHT scores 3.5-12.7% higher than the best long-context baselines on BEAM probing questions.
When should an agent team actually build structured memory?
Start building structured memory when your agent's conversation history exceeds 100K tokens per user or session, or when you need facts to persist across separate sessions at all. Below 100K tokens with no cross-session requirements, a well-managed context window may be sufficient - but the BEAM numbers suggest you should instrument for degradation before assuming it is fine.