How AI Agent Memory Actually Works

Most AI agents running in production right now have the memory of a goldfish. Not because the models are weak - because nobody wired up a memory layer beyond the context window. That single oversight is what causes a support agent to ask a user for their account tier on the third conversation in a row.

Here is what is actually going on under the hood, and why it matters more than context window size.

The four memory types, and what each one actually stores

AI agents use four types of memory drawn from cognitive science - in-context (working) memory, episodic memory, semantic memory, and procedural memory - formalised for LLMs in the CoALA framework (Princeton, arXiv:2309.02427). This is not vendor taxonomy. The cognitive-science lineage behind these distinctions is genuinely standard: the working-memory model is associated with Baddeley, the episodic/semantic split with Tulving, and the declarative/procedural distinction with Squire. The reason this matters for an engineering context is simple: these are not vendor-invented categories.

Here is what each tier stores and where it lives:

Working memory is fast - zero retrieval latency, because it is already in context - but volatile and capacity-limited. Everything else has to be fetched.

The distinction that trips teams up most: semantic memory stores declarative facts and relationships - user preferences, entity properties, domain knowledge. Unlike episodic memory, semantic memory is largely atemporal - it represents what the agent believes to be currently true. Episodic memory, by contrast, is the timestamped log of what actually happened. You need both. An agent that only has semantic memory cannot tell you when a fact changed. An agent with only episodic memory cannot efficiently generalise across sessions.

The retrieval mechanics - and where the latency actually comes from

Here is the concrete scene. An agent in a Slack channel gets asked: "What's our SLA for enterprise customers?" The agent has four options, in order of cost:

1. Hit working memory. If the SLA was mentioned earlier in this conversation, it is already in context. Zero retrieval cost.
2. Query episodic memory. Search past conversation logs for threads where SLA was discussed. Vector search required.
3. Query semantic memory. Pull the extracted fact "enterprise SLA = 4-hour response" from the knowledge graph. Faster than episodic if facts were previously distilled.
4. Fall through to RAG. Hit the doc store and search raw text. Slowest, most token-hungry.

The trap is option four. The full-context approach - dumping complete conversation history into the prompt - delivers the highest accuracy ceiling, but at a cost that makes it categorically unusable in production.

Mem0's research puts a concrete number on this: their token-efficient algorithm runs at under 7,000 tokens per retrieval call, roughly a quarter of the 25,000-plus tokens a full-context approach spends.

The other latency source nobody budgets for: reranking. Reranking adds accuracy but stacks another 50-150ms on top of your vector search.

Vector database retrieval alone runs 200-500ms before the embedding model, reranking step, and LLM invocation. That compounds if your agent makes multiple retrieval calls per reasoning step.

How memory gets written - the part most guides skip

Reading is the easy half. Writing is where production agents fall apart.

Working memory is written implicitly by the act of running. Episodic memory is written automatically by logging. Semantic memory is written by an extraction step - a background process that distils episodes into facts.

That extraction step is expensive and imprecise. During conversations, the memory layer extracts facts and stores them in a vector database indexed by user, session, and agent identifiers. At the start of a new session, relevant memories are retrieved using semantic similarity, keyword matching, and entity matching, then injected into the context window before the model responds. Only the most relevant facts surface, keeping token usage low and retrieval precise.

The non-obvious engineering problem: write-heavy patterns matter more in 2026 than a year ago because agent memory workloads look nothing like classic RAG. For continuous-write agent memory under 10M vectors, pgvector with IVFFlat handles writes cheaply and inherits Postgres's WAL for durability. Classic RAG optimises for reads; agent memory optimises for both. Agent memory means frequent writes, iterative retrieval, fact extraction, and entity resolution across conversation turns. Read-optimised HNSW indexes degrade under this workload.

What happens when facts conflict? When facts conflict, Mem0 self-edits rather than appending duplicates. Letta's approach is different: Letta/MemGPT implements OS-style virtual context management, where the LLM itself acts as memory manager via tool calls, paging content between a small main context and an unbounded archival store. Consolidation is agent-driven. Both work. The trade-off is that Letta's approach costs tokens. Letta beats vector-only RAG when memory has to evolve - write, rewrite, consolidate - not just be retrieved. The cost is real: more tokens per turn and slower loops, because the agent spends some of its reasoning budget on memory housekeeping.

The procedural tier: highest leverage, worst tooling

Procedural memory is where agent performance compounds. CoALA splits this into implicit procedural memory - the skills baked into the LLM's weights - and explicit procedural memory - the agent's own code, prompts, and learned rules. The implementation form for the explicit half is system prompts, playbooks, skills, and validated runbooks.

Here is the gap the vendors do not advertise: mem0's State of AI Agent Memory 2026 report describes the tooling for managing procedural memory specifically as "still early-stage." That means most frameworks give you rich primitives for episodic and semantic retrieval, and then effectively tell you to manage your own playbooks. The agent learns that a certain customer needs tickets routed to Tier 2 - but has no structured place to put that rule so it reliably applies in future sessions.

Key remaining challenges include temporal abstraction at scale, modelling cross-session structure so memories evolve rather than overwrite, and handling memory staleness when previously retrieved facts become incorrect after circumstances change.

The practical implication: if you are building an agent that should get better over time at your team's specific workflows, invest in explicit procedural memory now. Write the playbooks. Give the agent a mechanism to propose updates to them. The ecosystem will not hand you this for free.

A teammate like Beagle sits at the working-memory end of this stack by design: every message it handles in Slack is scoped to the active channel thread, with retrieval triggered by a real human request. That bounded scope is what makes the draft-and-approve model safe - a human reviews before anything posts, which means bad retrievals surface before they cause damage.

AI agent memory: common questions

What is the difference between agent memory and RAG?

RAG retrieves documents at query time and injects them into context. Agent memory persists, updates, and forgets facts across sessions - it is stateful where RAG is stateless. A RAG system answers the same question the same way every time; an agent with memory can track that a user's situation changed.

Why not just use a large context window instead of external memory?

Full-context approaches spend 25,000-plus tokens per retrieval call and incur proportional inference cost at every turn. Wider context windows mean more tokens for the LLM to process, which compounds total response time well past any reasonable agent latency budget. External memory retrieves only the relevant slice - typically under 7,000 tokens - and keeps inference cost flat regardless of conversation history length.

Which frameworks handle agent memory out of the box?

Most major frameworks, including Letta, Mem0, and LangChain, use the CoALA taxonomy as their foundation.

Five frameworks dominate: Mem0 (production-grade cross-tool layer), Letta/MemGPT (stateful agent framework with academic roots), Zep (temporal knowledge graph), MemPalace (local-first verbatim storage), and claude-mem (Claude Code IDE plugin). They are not interchangeable - Mem0 optimises for fast semantic extraction, Letta for agent-controlled consolidation, Zep for temporal reasoning.

How does memory consolidation work?

Consolidation is the process of merging or deduplicating episodic records into higher-level semantic facts. Local consolidation focuses on fine-grained updates involving highly similar memory fragments: each new memory retrieves its top-K most similar candidates, and an LLM decides whether merging is appropriate, reducing the risk of incorrect generalisation. Without consolidation, appending every interaction to a vector store eventually produces retrieval noise, context dilution, and latency spikes. Memory consolidation is what prevents this.

What is procedural memory and why does it matter for teams?

Procedural memory stores the agent's learned rules and playbooks - not facts about the world, but facts about how to act. It is the tier that lets an agent get measurably better at a team's specific workflows over time, rather than starting fresh each session. The tooling for it is the least mature of the four tiers, which means it requires the most deliberate design investment.