How AI Agent Memory Works: The Machinery Behind the Recall

Your AI assistant surfaces a preference you mentioned three weeks ago - your team's staging environment name, the quirky abbreviation nobody outside the team uses - and drops it into a reply without prompting. It feels like memory. It isn't. The model was stateless when you sent your message, and it will be stateless again the moment it replies.

What happened in between is what this post is about.

What "agent memory" actually means

AI agents read and write memory through a layered system built on top of a stateless language model. Short-term memory holds the current conversation inside the context window and disappears when the session ends. Everything else - anything you want to persist across sessions - has to be explicitly extracted, stored somewhere outside the model, and retrieved back into the context window on the next call.

In-context memory is the context window - everything the LLM processes in a single inference: system prompt, conversation history, retrieved chunks, and tool outputs. It is the only memory the model directly reasons over. All other memory types must be retrieved into the context window to influence generation.

That distinction is the foundation. The model has no persistent self. The agent loop around it does.

The CoALA framework (Princeton, arXiv:2309.02427) formalised this taxonomy from cognitive science for language model agents in 2023. It gave the field a clean vocabulary, and that vocabulary has stuck.

The four memory types and how they're stored

Each type is not just a category - it maps to a different storage backend, a different read/write pattern, and a different failure mode.

Working memory is the active context window. It is the agent's active context window: everything it can see in the current moment. Working memory is temporary and bounded by the size of the context window, but it is where all the other memory types converge. Relevant facts from semantic memory, pertinent history from episodic memory, and applicable rules from procedural memory are all retrieved and assembled here before the agent responds. Think of it as RAM. Fast, finite, gone when the session ends.

Episodic memory is the record of what happened. It is the record of what has actually happened - past interactions, their sequence, their outcomes. An AI agent with strong episodic memory knows not just what a user prefers, but what they have tried, what worked, what did not, and what was promised in earlier conversations. This is the memory type most critical for long-running tasks and multi-step workflows where continuity is everything.

Episodic memory captures specific past experiences with temporal details. Store this using vector databases for semantic search and event logs for ground truth.

Semantic memory is the knowledge base. Semantic memory stores general facts and definitions - what things are, independent of when or where they were learned. Episodic memory stores specific past events tied to time - what happened, when, in which session.

Unlike episodic memory, which records individual interactions, semantic memory extracts and preserves key information - such as turning a past interaction about a peanut allergy into a permanent fact. In a work context: your team's naming conventions, the definition of "P0", which Slack channel owns incidents. That's semantic memory.

Procedural memory covers how to do things. Procedural memory captures how to perform tasks: workflow steps and decision points. Store using workflow databases and vector databases for similar task retrieval.

The read/write asymmetry nobody talks about

Here is the part most explanations skip. Writes are slow and batched, since they require embedding, extraction, and indexing. Reads are fast and constant, pulling relevant memory into context on every turn.

When a relevant memory exists, the retrieval system must surface it within a strict latency budget, typically under 200 milliseconds.

RAG generally incurs a per-query latency of 1-2 seconds (plus memory construction), while long-context agents require around 5 seconds per query.

Writes are where the real cost sits. Traditional memory pipelines run three sequential LLM calls per new memory: extract the raw fact, check for conflicts, then update or merge. This is expensive and adds latency on every write. One research paper (MemDelta, arXiv:2606.29914) put concrete numbers on this: a Mem0-style extraction strategy with 1,000+ LLM calls per 50-session ingestion costs $0.50+ at write time, versus $0.01 for verbatim RAG with zero LLM calls during ingestion.

Write-path cost can consume over 80% of total agent execution time, yet it is rarely reported in memory benchmarks.

The non-obvious consequence: if your agent is slow, the problem is probably not retrieval. It's the write path.

The token number comes from Mem0's April 2026 benchmark report: their token-efficient memory algorithm achieved a LoCoMo score of 92.5 at roughly 6,956 tokens per retrieval call. The full-context baseline, which injects the complete conversation history into the window, required approximately 26,000 tokens per conversation to achieve lower scores.

That gap - 6,956 vs. 26,000 tokens - is the economic argument for selective memory over a long context window. See the earlier field note Does a Bigger Context Window Replace Agent Memory? for the full comparison.

A concrete walk-through: the Slack question

Put it together with a real scenario. An agent in a Slack channel gets asked: "What's the status of the Northstar migration?"

Here is every memory layer that fires before the model writes a single token:

1. Working memory loads: the current message, the system prompt, any tool schemas available.
2. Episodic retrieval: the memory system embeds the query, runs vector similarity search against stored past events, and pulls the top-k chunks - "last week, @maya said the migration would slip to Q3", "three days ago, the infra team posted a blocker in #platform".
3. Semantic retrieval: a key-value or vector lookup adds standing facts - "Northstar = internal name for the payments service rewrite", "migration owner = @maya".
4. Procedural memory: any rules encoded in the system prompt fire - "always cite sources", "tag the DRI in any status update".
5. All of that gets assembled into the context window. The model generates a reply grounded in retrieved context it couldn't have known from training.

When users ask an LLM a question, the AI model sends the query to another model that converts it into a numeric format. The numeric version of the query is sometimes called an embedding or a vector. The embedding model then compares these numeric values to vectors in a machine-readable index of an available knowledge base. When it finds a match or multiple matches, it retrieves the related data, converts it to human-readable words and passes it back to the LLM.

The model never "knew" any of this. The memory system assembled it, just in time, for that single inference call.

Which framework to pick

Vector-first frameworks like LangMem and SuperMemory use similarity-based retrieval and are simpler to reason about, primarily solving personalization.

Vector + Graph systems like Mem0, Zep, and Cognee handle entity relationships and structured knowledge, with varying emphasis on temporal reasoning - though Mem0 gates graph features behind Pro.

The accuracy gap between architectures is real. Mem0 is broader and easier to adopt; Zep is more accurate for temporal queries. On LongMemEval using GPT-4o, Zep scores 63.8% vs. Mem0's 49.0% - a 15-point gap driven by Zep's temporal knowledge graph, which stores fact validity windows rather than timestamped snapshots.

LangMem is primarily designed for LangGraph and offers limited value outside it. LlamaIndex Memory is tied to LlamaIndex. If there's any chance you'll change frameworks, start with a standalone memory system.

One genuinely non-obvious trap: retrieval quality degrades in production not because search is bad, but because irrelevant past state keeps outranking fresh context. More memory is not always better memory. Systems need write policies, expiration windows, and conflict resolution - not just a growing vector store.

How AI agent memory works: common questions

What is agent memory in simple terms?

Agent memory is an external storage layer that compensates for the fact that language models are stateless. Between turns, the agent writes selected facts to a database; at the start of each new turn, it retrieves the most relevant facts back into the context window. The model never stores anything - the surrounding system does.

What is the difference between episodic and semantic memory in AI agents?

Semantic memory stores general facts and definitions - what things are, independent of when or where they were learned. Episodic memory stores specific past events tied to time - what happened, when, in which session. Semantic memory is queried for "what does revenue mean"; episodic memory is queried for "did this agent run this report before."

Why is agent memory write latency a problem?

Traditional memory pipelines run three sequential LLM calls per new memory: extract the raw fact, check for conflicts, then update or merge. This is expensive and adds latency on every write. Single-pass extraction runs one LLM call to extract a structured fact and add it to the store, with conflict resolution deferred to retrieval time. This cuts write-time LLM calls by 60 to 70 percent without meaningfully degrading memory quality.

Does a bigger context window make agent memory unnecessary?

No. The purpose of agent memory is to avoid loading the full conversation history or user profile into the prompt on every turn. Instead, the application stores facts externally and retrieves only the subset that is relevant to the current task. Larger windows raise the ceiling but don't fix the token cost or the signal-to-noise problem of dragging all history into every call.

How do I choose between Mem0, Zep, and LangMem?

For managed, drop-in personalization memory, Mem0 leads on community size and compliance posture. For temporal reasoning, Zep's Graphiti engine scores 15 points higher on LongMemEval. Teams on LangChain should evaluate LangMem first. Long-running agents benefit from Letta's tiered memory model. Start with the use case, not the feature list.