Your agent aces its demo on Monday. A returning user comes back on Thursday with a different job title, a closed deal, or a changed preference. The agent confidently repeats what it knew last week. That is not a model problem. It is a memory problem - and across production agent systems, this is the defining Day-2 failure: the demo works because the whole conversation fits in one context window, and production breaks because real users come back across sessions, days, and topics.
The benchmark that has started separating real memory systems from demo-grade ones is LongMemEval. The scores vary by 40+ percentage points across leading frameworks. Understanding why the gap is that large tells you more about how to build a production memory layer than any vendor feature page.
What LongMemEval actually tests
LongMemEval focuses explicitly on the chat-assistant memory case. It defines five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. The full eval set is 500 curated questions. The temporal reasoning and knowledge-update categories are the ones that kill naive implementations - they require knowing not just what a fact says, but when it was valid.
LongMemEval_S has chat histories of roughly 115,000 tokens spread over about 40 sessions per user. LongMemEval_M extends that to roughly 500 sessions per chat history - the regime where naive context-stuffing breaks down completely.
Two other benchmarks round out the field: LoCoMo covers 1,540 questions across single-hop, multi-hop, open-domain, and temporal recall; BEAM evaluates at 1M and 10M token scales. But for most teams building session-persistent agents, LongMemEval is the most directly applicable. The practical takeaway for agent builders is to read benchmark scores in pairs: pair an accuracy number with its token cost; pair a single-session score with a multi-session one; pair a context-window probe with a memory eval. The pair is what tells the truth.
The 15-point gap between vector and graph architectures
This is where the architectural split between frameworks becomes concrete.
On LongMemEval, Zep's Graphiti backend scores 63.8% with GPT-4o while Mem0 lands at 49.0%. That is a 15-point gap on the exact capability - remembering facts that change over time - that most production agents need and most demos never test.
The root cause is structural. Mem0 uses vector similarity retrieval. Temporal queries require knowing when a fact was valid, not just what it currently says. Vector search returns the most semantically similar stored memory - which is the correct approach for "what does this user prefer?" but the wrong approach for "is this user's employer still the same company they mentioned in session 3?"
Zep's Graphiti engine stores facts with valid_at and invalid_at timestamps on every node and edge, meaning it knows not just what an agent remembered, but when that was true.
That architectural choice - knowledge graph over flat vector store - is what produces the score gap. It also explains why graph-native memory costs more: Zep starts at $25/month for graph features, while the equivalent Mem0 graph tier is gated at $249/month.
The current leaderboard, simplified:
| Framework | LongMemEval overall | Architecture | Self-hostable |
|---|---|---|---|
| Hindsight | 91.4% (Gemini-3 Pro) | Hybrid | Yes (free) |
| Mem0 (new algorithm) | ~93.4% | Vector + selective edit | Yes (Apache 2.0) |
| Zep / Graphiti | 71.2% | Temporal knowledge graph | Graphiti only |
| Letta | 83.2% | OS-inspired tiered memory | Yes |
| Mem0 (GPT-4o, older) | 49.0% (temporal sub-task) | Vector similarity | Yes |
A few notes on reading that table honestly. Mem0's new token-efficient algorithm, published April 2026, closed the gap significantly - the older score of 49.0% on temporal retrieval rose to 93.4% per Mem0 Research. Independent verification of that number is still limited; treat it as a self-reported claim until third-party evals catch up. Hindsight's 91.4% is real, but it reaches that score with Gemini-3 Pro
- swap in a cheaper model and the number drops.
Most memory benchmarks fit entirely in modern context windows, making memory systems unnecessary for those specific tests. If a vendor's eval doesn't specify the session count and context length, it may be measuring something a long-prompt baseline would also pass.
The staleness problem nobody benchmarks well
Getting recall right is the tractable part. The harder problem is confident wrongness - when a highly-retrieved memory is no longer accurate.
Memory staleness occurs when a highly-retrieved memory about a user's employer is accurate until they change jobs, at which point it becomes confidently wrong. Decay handles low-relevance memories. Staleness in high-relevance memories is a harder, open problem.
This matters more than the benchmark scores suggest. A memory system that scores 71% on LongMemEval but surfaces stale facts confidently is worse in production than one that scores 60% but abstains when uncertain. LongMemEval does have an abstention category, but a system that scores well on accuracy but requires 26,000 tokens per query is not production-viable
- and one that answers confidently with outdated facts is not trustworthy.
The two other open problems the research consensus flags:
cross-session identity resolution (the memory model assumes a stable user_id, but anonymous sessions, multi-device users, and mixed auth flows break that assumption); and resolving whether two interactions came from the same person is an unsolved identity problem at the memory layer.
A teammate like Beagle handling team queries in Slack runs into both: the same person might open threads from a browser, a mobile client, and a bot mention, and their role or project may change between conversations.
Beagle in action#product, Tuesday 10:22amHow to pick a framework before you have your own evals
The benchmark table is a start. The real question is which failure mode you can least afford.
If your agent tracks facts that change (user role, project status, deal stage): start with Zep / Graphiti. The temporal knowledge graph architecture is specifically built for this, and the 15-point LongMemEval gap is the empirical evidence.
If your agent needs personalization across a fixed user profile: Mem0's new algorithm is credible - the April 2026 token-efficient update is worth testing if you were previously priced out by the graph tier.
If you're running inside LangChain or LangGraph: LangMem integrates natively with zero additional infrastructure.
If your agent needs to keep improving across runs (coding agents, long-horizon research tasks): Letta's tiered memory model is built for long-running agents.
If air-gapped deployment is a hard requirement: Cognee and Memori are the options; neither currently advertises SOC 2, which matters for healthcare, finance, or any enterprise procurement that gates on certifications.
One thing the framework comparisons mostly skip: governance. The differences between frameworks show up at scale: temporal accuracy, multi-agent coordination, compliance posture, and whether the framework can be grounded in your actual data estate rather than just session history. A memory layer that stores facts without lineage or audit trail will eventually create a different kind of problem than the ones LongMemEval measures.
Agent persistent memory: common questions
What is LongMemEval and why does it matter for agent memory?
LongMemEval is a 500-question benchmark that tests AI agent memory across five categories: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. It matters because it covers the scenarios - facts that change across sessions - that most agent demos never test and most production deployments eventually hit.
How does Zep compare to Mem0 on benchmarks?
On GPT-4o, Zep scores 63.8% and Mem0 scores 49.0% on LongMemEval, a 14.8 percentage point gap in Zep's favor. The gap comes from architecture: Zep's Graphiti engine stores timestamps on every fact, making temporal queries tractable in a way flat vector retrieval is not. Mem0's April 2026 algorithm update claims to close most of that gap, but independent verification is still limited.
What is the memory staleness problem in AI agents?
Memory staleness is when a highly-retrieved stored fact was once correct but is now wrong - a user's old employer, a superseded policy, a closed project. Unlike low recall (which benchmarks catch), staleness produces confident wrong answers, which is often worse. No current benchmark fully penalizes this; the abstention category in LongMemEval partially addresses it.
When should I use a knowledge graph memory layer instead of a vector store?
Use a knowledge graph - Zep / Graphiti, Cognee - when your agent needs to track how facts change over time, not just which facts are semantically similar to the query. If your memory only needs to answer "what does this user prefer?" vector retrieval is sufficient. If it needs to answer "what was this user's role last quarter, and has it changed?" you need timestamps on nodes and edges.
Do larger context windows make persistent memory frameworks unnecessary?
Frontier models in 2026 have 128K-1M context windows , which handles short-to-medium session histories inline. But LongMemEval_M extends to roughly 500 sessions per chat history - the regime where naive context-stuffing breaks down completely. At that scale, token cost alone makes full-context approaches unworkable; a selective memory layer that writes salient facts and injects them at retrieval time is the only cost-viable option.