Dumping full conversation history into the prompt is the first thing every agent builder tries. The research shows why it breaks: a median response latency of 9.87 seconds, with one in twenty requests taking 17 seconds - at a token cost roughly 14 times higher than a selective memory approach.
That number comes from a 2026 benchmark comparing full-context versus retrieval-based agent memory at production load. The gap is big enough that "use a bigger context window" is the wrong mental model for memory. The right model is closer to database design: different data belongs in different stores, retrieved differently, at different costs.
Here's how it actually works.
What "agent memory" actually means (and why the context window is not enough)
Short-term memory - also called working memory - holds information only during a single conversation session; it resets when the session ends. That's the context window. It's fast, zero-latency, always coherent. But it is bounded and amnesiac: once you close the thread, everything in it is gone.
Long-term memory allows an agent to retain and reuse information beyond a single interaction. It is typically implemented outside the model using vector databases, document stores, or structured knowledge layers. The model doesn't hold this data; it reaches for it when needed.
The practical consequence of that distinction is what catches most builders off guard. A context window gives you coherence within one session. Long-term memory gives you continuity across sessions. You need both to build an agent that behaves like a knowledgeable colleague rather than a goldfish with good grammar.
Large context windows delay but do not fix memory failures. The model still loses the thread once the history exceeds the window; it just takes longer to hit that wall.
The four types of memory and what each one stores
Within long-term memory, more specialized forms - episodic, semantic, and procedural memory - determine how information is organized and used. Here's what each does in practice:
| Memory type | What it stores | Where it lives | Example |
|---|---|---|---|
| Short-term | Current conversation turns, active task state | Context window | "The user just asked about Q3 churn" |
| Episodic | Past interactions and their outcomes | Vector DB / log store | "Last Tuesday this query failed because the data wasn't loaded yet" |
| Semantic | Facts, rules, domain knowledge | Vector DB / knowledge graph | "Our refund policy is 30 days, no questions asked" |
| Procedural | How to do things, learned workflows | Often baked into prompts or fine-tuning | "Always check auth before querying the billing API" |
Episodic memory tells the agent "Last Tuesday, when we tried approach X with client Y, it failed because of Z." Semantic memory tells the agent "Approach X generally works best when conditions A and B are present." Both are essential, but they serve different cognitive functions.
The common mistake is conflating semantic and episodic memory - storing conversational logs (episodic) in the same flat vector store as facts (semantic), then querying them the same way. The recall quality suffers because similarity search over "what happened" and "what is true" pulls from very different distributions.
How retrieval actually works: the read path
When an agent needs to answer a question, the memory read path runs before the model call, not inside it. Retrieval-based memory adds read-path latency because it runs a vector search before each inference call. That's a real cost - but it's cheaper than the alternative.
The read path in a production agent looks like this:
Embed the query - the incoming message is converted to a vector using an embedding model.
Search the store - a vector similarity search finds the k most relevant memory chunks. At a 99% recall threshold, Postgres with pgvector and Qdrant both hit sub-100ms maximum query latency.
Rerank (optional) - a lightweight cross-encoder reranks the results for relevance. Reranking adds accuracy but stacks another 50-150ms on top of your vector search.
Inject into context - only the retrieved chunks go into the prompt, not the full history.
Run the model - the model now has relevant memory without paying for irrelevant tokens.
The token math is concrete: 6,956 tokens per retrieval call on LoCoMo vs roughly 26,000 for full-context is a real difference on your inference bill.
The full-context approach delivers the highest accuracy ceiling, but at a cost that makes it categorically unusable in production: a median latency of 9.87 seconds and a p95 latency of 17.12 seconds, meaning one in twenty users waits 17 seconds for a response, at a token cost roughly 14 times higher than selective memory approaches.
The write path: where the real complexity lives
Reading memory is well-understood. Writing it cleanly is the harder problem.
Traditional memory pipelines run three sequential LLM calls per new memory: extract the raw fact, check for conflicts, then update or merge. This is expensive and adds latency on every write.
Systems requiring LLM-mediated entity extraction - including Mem0's graph variant and Zep - consume tokens and incur non-trivial latency at every write operation. Zep invokes two or more LLM calls per write, resulting in ingestion latencies of approximately 3 seconds. For a support agent handling 1,000 messages a day, that compounds into real cost fast.
The emerging answer is deferred conflict resolution: extract a clean atomic fact in one LLM call, store it immediately, and reconcile conflicts asynchronously. This cuts write-time LLM calls by 60 to 70 percent without meaningfully degrading memory quality.
A second non-obvious problem: none of the evaluated systems prune or forget by default, so footprint grows monotonically under default behavior; bounding fleet storage requires an independent forgetting policy. An agent that runs for six months will accumulate stale episodic memories - old ticket resolutions, superseded policies, deprecated API endpoints - that actively hurt retrieval quality unless you build a TTL or relevance-decay mechanism.
Beagle in action#support-ops, 11:22amHow this connects to the tool-calling loop
Memory doesn't sit in isolation - it's one node in the broader agent execution cycle. A typical call looks like: receive message → retrieve relevant memory → build context → call model → model emits either a response or a tool call → your code executes the tool → result feeds back into context → model continues.
It's important to emphasize that when using function calling, the LLM itself does not execute functions. Instead, it identifies the appropriate function, gathers all required parameters, and provides the information in a structured JSON format. Memory retrieval is architecturally identical: the model signals it needs a memory lookup, your scaffolding runs the vector search, and the result goes back to the model as another context message.
A simple breakdown: model call takes 500ms to 2s, tool call 200ms to 1s, memory retrieval 100ms to 300ms. An agent that feels "instant" in a demo can easily take several seconds in production.
The practical implication: if your agent does memory retrieval plus two tool calls in one turn, you're already stacking 800ms-3.5s of non-model time before the LLM even generates its first output token.
AI agent memory: common questions
What is the difference between agent memory and the context window?
The context window is short-term memory - it holds the current conversation and resets at session end. Agent memory is the broader system: it includes external vector stores, knowledge graphs, and log databases that persist across sessions. The context window is where the model reads; external memory is where the agent writes and retrieves over time.
Why not just use a large context window instead of a memory system?
Token cost and latency. Passing full conversation history into every call costs 3-14× more tokens than selective retrieval, and produces median response times nearly 10 seconds at production load. Selective memory retrieval keeps both token spend and latency manageable as conversation history grows.
What is episodic memory in an AI agent?
Episodic memory stores specific past interactions and experiences - like a conversation log the agent can recall later. Unlike semantic memory (general knowledge), episodic memory captures the when and how of past events. In practice it's stored as timestamped records in a vector or graph database and queried by semantic similarity to the current task.
How does an agent actually retrieve a memory?
The agent embeds the current query into a vector, runs a similarity search against the memory store, optionally reranks the top results, and injects the most relevant chunks into the prompt context before the model call. The model never "sees" the full memory store - only the retrieved slice.
Do agents forget old memories automatically?
Not by default. None of the commonly evaluated systems prune or forget by default, so footprint grows monotonically under default behavior; bounding fleet storage requires an independent forgetting policy. You need to build or configure TTL rules, relevance decay, or periodic consolidation to prevent stale memories from degrading retrieval quality over time.