How AI Agent Memory Works, From Context Window to Vector Store

An AI agent with no memory layer asks the same onboarding question on day three that it asked on day one. It answered it yesterday, but stateless LLM calls can parse text and generate drafts - they cannot remember. The context window resets. The interaction history evaporates. What is left is a fast, expensive autocomplete that cannot learn.

This is the gap that the agent memory layer is designed to fill, and it is where most of the interesting engineering in AI stacks is happening right now. Understanding how AI agent memory works requires pulling apart four distinct things that get conflated: the context window as working memory, episodic memory for past interactions, semantic memory for facts and knowledge, and procedural memory for how to act. None of those is the same problem, and none of them uses the same storage.

In-context memory: the only memory the model actually reads

In-context memory is the context window - everything the LLM processes in a single inference call: system prompt, conversation history, retrieved chunks, and tool outputs. It is the only memory the model directly reasons over. All other memory types must be retrieved into the context window to influence generation.

Think of it as RAM. Fast. Directly accessible. But short-term memory in this sense is the active context window - the information currently in scope during a single model call. It is fast and directly accessible but disappears when the session ends.

The implication is architectural, not cosmetic. You can throw every interaction into the context window and pay for a model to reason over 100,000 tokens every call, or you can build a layer that decides what the model actually needs to see. Designing an agent's memory is essentially context engineering: determining which tokens enter the context window and how they're organized.

Mem0, benchmarked at ECAI 2025, attains 91% lower p95 latency and saves more than 90% token cost compared to the full-context method. That gap is why selective memory is not an optimization - it is table stakes for anything running at production scale.

Episodic memory: what happened, when, and with whom

Episodic memory stores past interactions and experiences - what happened, when, and with whom. A customer support agent with episodic memory knows that a specific user submitted a refund request last month and escalated it.

Implementation is concrete: episodic memory for an AI agent is usually implemented as a memory module connected to the agent's decision-making logic. Each time a significant event occurs, the agent creates a new entry in memory. Each memory entry may include metadata such as timestamp, entities involved, results, and potentially embeddings for similarity searches.

When a new input arrives, the agent does not scan a spreadsheet. The incoming query is encoded into a vector representation, which is then compared against vectors stored in the memory database to locate similar episodes. The most relevant memories are returned and inserted into the agent's reasoning flow.

The three factors that govern what gets surfaced during retrieval were formalized in the Generative Agents paper (arXiv:2304.03442): recency (how recent was it?), relevance (how semantically similar is it to the current context?), and salience (how surprising or important was it when it occurred?). A memory that scores low on all three gets buried; one that is recent, relevant, and surprising gets injected.

One real failure mode: timestamps. OpenAI's built-in memory performed poorly on time-sensitive questions because it consistently failed to attach timestamps to stored memories, making chronological questions nearly impossible. Temporal structure is not optional for episodic memory; it is the point.

Semantic memory: what the agent knows, not what it experienced

Semantic memory stores facts and knowledge - things the agent should generally know. Think product documentation, company policies, or a knowledge base. This is typically implemented using vector embeddings stored in a vector database or a vector extension on top of Postgres, such as pgvector. The agent retrieves relevant chunks at query time using similarity search.

The distinction from episodic memory matters. Unlike episodic memory, which records individual interactions, semantic memory extracts and preserves key information - such as turning a past interaction about a peanut allergy into a permanent fact like "User Allergy: Peanuts." The episodic event is specific; the semantic fact is generalised and reusable.

A crucial part of building intelligent agents is converting episodic memory into semantic memory. This process involves identifying patterns across past interactions and distilling them into reusable knowledge. In practice, this is often done by a separate summarisation step or a background consolidation agent - not the main LLM during inference.

In LLM agents, semantic memory exists both in pre-trained model weights and in external storage for organisation-specific facts the model was not trained on. The model already knows what a contract is; it needs external retrieval to know what your specific NDA terms say.

Procedural memory and the full stack

Procedural memory encodes how to do things - workflows, decision logic, and action patterns. In agentic systems, this often lives in the agent's system prompt or as retrieved workflow definitions. It is the least exotic of the four types and the easiest to get wrong: a system prompt that grows to 8,000 tokens of "how to act" instructions leaves almost no room for the information the agent actually needs to act on.

MemGPT, which became the Letta framework, treats context windows as a constrained memory resource and implements a memory hierarchy similar to operating systems. The system provides function calls that allow the LLM to manage its own memory autonomously. Agents can move data between in-context core memory (analogous to RAM) and externally stored archival and recall memory (analogous to disk), creating an illusion of unlimited memory while working within fixed context limits.

Letta positions this transparency as essential for debugging, claiming 90% of agent failures can be traced to context window issues. Context budget management allows developers to set maximum token limits for the context window. When this budget is exceeded, Letta's summarisation system automatically compacts older information to stay within limits while preserving essential learned knowledge.

The practical stack for a production agent ends up looking like this: in-context memory holds the current conversation and the retrieved chunks; a vector database (Pinecone, Qdrant, Weaviate, or pgvector depending on your infra) holds semantic knowledge and embedded episodic logs; a relational store like Postgres holds structured event history; and a consolidation job periodically promotes episodic traces to semantic facts. No single storage mechanism solves every memory problem. In-memory caches, relational stores, key-value systems, graph structures, and vector search all have a role.

One honest caveat: vendor benchmark numbers and production behaviour diverge. A system that scores well on accuracy but requires 26,000 tokens per query is not production-viable. A system with low latency but poor recall is not useful. And independent testing shows real gaps - Mem0 v0.8.2 scores 91.6 on LoCoMo under vendor benchmarks, while independent production testing at 50K sessions returns 49% effective accuracy after 30 days, a 32-point gap from stale data and entity contradiction. That is not a reason to avoid memory layers; it is a reason to instrument them.

What actually breaks in production

The four failure modes worth knowing before you build:

Stale facts. Zep's temporal knowledge graph timestamps every fact, so if a user says "I moved from London to Tokyo," Zep models the state change instead of returning both cities as current. Most simpler stores do not do this.

Context collapse. LLMs favour short answers and drop nuance. Iterative summarisation erodes details over time. A consolidation loop that summarises too aggressively will silently lose the precise facts that make memory useful.

Write consistency. Real-world teams at scale have reported indexing reliability issues - memories not being added consistently and context recall failures under load. Memory is infrastructure; it needs monitoring.

Token budget creep. Semantic memory stores grow. A 500-document knowledge base at ingestion becomes 5,000 chunks, and retrieval quality degrades if you are not pruning and re-ranking. Adding a vector database does not guarantee good retrieval. Test your retrieval with real queries against real data early.

A teammate like Beagle, running inside Slack or Teams, faces these same constraints in miniature. The questions it answers - "what did we decide in last week's standup?", "where does the Q3 budget doc live?" - are episodic retrieval problems. Getting the memory layer right is what separates a bot that impresses in a demo from one that is still useful at month three.

The core insight is simple even if the implementation is not: LLM agents increasingly operate in settings where a single context window is far too small to capture what has happened, what was learned, and what should not be repeated. Memory - the ability to persist, organise, and selectively recall information across interactions - is what turns a stateless text generator into a genuinely adaptive agent.