How Does AI Agent Memory Actually Work?

Ask a customer-support agent what a user said three sessions ago and it goes blank. Not because the underlying model is weak - the model may be excellent - but because nothing connected the dots between sessions. That gap is an architecture problem, not a capability one, and it has a name: the agent had no memory layer.

AI agents use four memory types drawn from cognitive science - in-context (working) memory, episodic memory, semantic memory, and procedural memory - formalised for LLMs in the CoALA framework from Princeton. Understanding what each one stores, where it lives physically, and how retrieval works is the difference between an agent that behaves coherently across sessions and one that starts from scratch every time.

The four types, and where data actually lives

The taxonomy sounds academic until you think about what breaks when a type is missing.

Short-term memory is inside-task working memory. Long-term memory divides into three types: semantic memory, which contains the agent's world knowledge; procedural memory, which involves rules or procedures that may reside implicitly in the LLM's weights or be explicitly defined as guidelines; and episodic memory, which records task-specific experiences.

Working (in-context) memory is the context window itself. It is fast - zero retrieval latency, because the information is already in context - volatile, lost when the context is cleared, and capacity-limited, bounded by the token limit. For most agents, this is the only "memory" being used. Everything else is bolted on.

Episodic memory is the raw event log. It preserves sequences of events as they happened: full conversations, task trajectories, ordered observations. Unlike semantic memory, it keeps narrative context and temporal flow. In practice, episodic memory often looks like a message store or indexed conversation chunks, sometimes designed explicitly to store what was evicted from working memory.

Semantic memory is the agent's fact store - distilled knowledge rather than raw transcripts. User preferences, product facts, resolved ambiguities. User preferences can be distilled from episodic experiences as semantic memory, while navigation skills can be abstracted as procedural memory.

Procedural memory is the slowest-changing tier. For LLM agents, procedural memory is typically encoded in model weights, system prompts, and tool registries. It changes more slowly than other memory types and usually requires explicit updates - deployments, fine-tuning, or tool additions.

How Letta treats the context window like RAM

Letta (formerly MemGPT) takes the OS analogy further than any other production framework. Its architecture gives the LLM explicit control over a memory hierarchy: main context (working memory - the actual context window, managed like RAM), archival memory (external storage for long-term facts), and recall memory (a searchable log of past conversation history).

The LLM itself decides when to push information from main context to archival ("paging out"), retrieve from archival to main context ("paging in"), search recall memory, or edit its own system prompt. These operations are implemented as function calls the model can invoke.

When the context fills up, Letta executes automatic compaction routines. The system acts as a virtual memory paging mechanism, compressing older historical message blocks into episodic summaries, committing them to Recall Memory, and evicting the raw tokens from the active window - keeping the agent focused on its primary objective while preserving an unbroken timeline of past decisions.

The cost of this approach is real. The agent spends some of its reasoning budget on memory housekeeping, adding tokens per turn and slowing loops. That trade-off makes Letta the right tool for long-running personal-assistant agents and multi-session products, and overkill for stateless single-turn pipelines.

The consolidation problem: moving facts between tiers

The hardest part of agent memory design is not picking a vector database. It is deciding when a raw interaction becomes a stored fact, and when a stored fact should be discarded because it is stale.

Over time, raw experience becomes more useful when summarised into stable knowledge. The Generative Agents paper popularised "reflection" mechanisms that periodically synthesise episodic memories into higher-level insights, stored as semantic memory and reused later.

Mem0 formalises a similar idea as a production-oriented pipeline: extract and consolidate important information from conversations into durable memories.

Mem0's token efficiency numbers are the most revealing benchmark in this space right now. Its new token-efficient algorithm hits 92.5 on LoCoMo and 94.4 on LongMemEval while averaging under 7,000 tokens per retrieval call. Full-context approaches on the same benchmarks use 25,000+ tokens. That is a 3-4x cost difference per query - which, at inference prices that bill by the token, maps directly to margin.

The category most affected by the additive architecture is knowledge update, at 93.6: older facts are preserved rather than overwritten, so a semantically similar prior fact can surface alongside a newer one. That is the staleness problem in concrete form: the memory layer is accurate but not always current.

What actually breaks in production

A system that only uses working memory (the context window) works fine for short conversations. It breaks in three recognisable ways as sessions grow:

• Blank-slate syndrome. Each new session starts cold. The agent asks for context the user already gave it last week.

• Stuffed context. Someone tries to fix blank-slate syndrome by passing the full chat history on every turn. Token costs scale linearly with session length. Attention quality degrades - the computational cost scales quadratically with the context window, and longer contexts show diminishing returns because models struggle to use the additional context size effectively.

• Stale facts. Extraction-based systems like Mem0 store facts once and retrieve them later, but they don't automatically overwrite conflicting information unless you build that logic. Older facts are preserved rather than overwritten, so a semantically similar prior fact can surface alongside a newer one.

The architectural answer is a deliberate consolidation schedule: episodic store captures every event, a summarisation step runs on a threshold (token count, time, or importance score), and distilled facts land in semantic memory. Periodically, the ongoing conversation is compressed into a concise summary stored in a dedicated summarisation store. Summarisation can be triggered by context-window token-limit thresholds, importance heuristics, a schedule, or exposed as a tool the agent can invoke at its discretion.

A teammate like Beagle handles the retrieval side of this in Slack and Teams - pulling relevant context from past threads before drafting a reply - so the human approves a draft that already has the history baked in, rather than reconstructing it by hand.

AI agent memory: common questions

What is agent memory in LLMs?

Agent memory is any mechanism that lets an LLM persist and retrieve information across turns or sessions. It divides into four types - working (in-context), episodic (event log), semantic (distilled facts), and procedural (rules and weights) - each stored differently and retrieved on different triggers.

Why does an AI agent forget between sessions?

Because the base LLM is stateless: it takes a prompt in and returns tokens out, with no persistence. Without an external memory layer attached to your agent, each new session starts from an empty context window. The solution is an episodic store that writes conversation records and surfaces them at the start of the next session.

What is the difference between RAG and agent memory?

RAG retrieves from a fixed document corpus before each generation call. Agent memory is a read-write store that updates as the agent acts - it captures preferences, past decisions, and interaction history that no static document contains. Unlike standard RAG, which passively retrieves before generation, a memory-augmented agent is proactive: the LLM decides if, when, and what to retrieve using tools.

What is episodic memory in an AI agent?

Episodic memory preserves sequences of events as they happened: full conversations, task trajectories, ordered observations. Unlike semantic memory, it keeps narrative context and temporal flow. It is valuable for audit trails, debugging, pattern analysis over conversations, and reflection loops that synthesise higher-level insights from experience.

How does an agent know what to store in memory?

It depends on the framework. Extraction-based systems (Mem0) run an LLM call after each turn to pull out atomic facts and embed them. Letta's self-paging model lets the agent decide via tool calls. Rule-based systems store on trigger conditions - a completed task, a user preference expressed explicitly, a context-window threshold crossed. There is no universal answer; the right trigger design depends on how often facts change and how sensitive stale retrieval is in your use case.