Agent Memory Has Three Layers, and the Middle One Is the Problem

The best model on the 2025 Episodic Memory Benchmark scores below 30% on chronological reasoning - the part where an agent tracks how facts change over time. That is not a capability shortfall. It is an architecture shortfall. The model can reason; it just was not given the right memory structure to reason against.

Agent memory is not a single thing. It is a stack of three distinct layers, each doing a different job, and the transitions between them is where systems break. Most explanations stop at "the agent remembers things from the conversation." That is describing only the top layer.

What working memory actually is

Whatever fits inside the current context window constitutes the agent's working memory. Baddeley's central executive model maps neatly: the LLM is the executive, the context window is the buffer, and both share the same bottleneck - limited capacity.

This means working memory is finite, volatile, and expensive per token. Anything not in the current context window does not exist, as far as the model is concerned. A support agent handling a long conversation will eventually push early messages out of its context. When that happens, archive too aggressively and you create "memory blindness" - the agent simply does not know that the critical fact exists somewhere in cold storage.

The practical implication: working memory is not a design choice. It is physics. Every agent has it, and every agent is bounded by it.

Episodic memory: where events land first

Episodic memory is the log. It records concrete experiences: individual tool calls, conversation turns, environment observations. In the Generative Agents work by Park et al., every observation lands in the episodic stream with a timestamp, an importance score, and an embedding for later retrieval.

The key properties an episodic store needs, per a 2025 position paper on long-term LLM agents, are: long-term storage (persistence beyond the session), explicit reasoning (the ability to reflect on memory content), single-shot learning (capturing information from single exposures without gradient updates), instance-specific memories (details unique to this occurrence), and contextual memories - who, when, where, why, bound to the content.

One trap: an agent that summarizes at write time collapses distinct episodes into semantic generalizations, destroying the episodic signal before it can be used. This is a common implementation shortcut. It feels efficient - fewer stored records - but it means the agent can no longer distinguish between two similar events that had different outcomes.

Semantic memory: where patterns live

Semantic memory stores abstracted, de-contextualized knowledge. An episodic fact like "the user corrected the date format on Jan 5, Jan 12, and Feb 1" may consolidate into the semantic record "user prefers DD/MM/YYYY." This consolidation is rarely automatic; most current systems require explicit prompting or heuristic triggers.

When it works, the four layers form a clean reasoning stack. After processing hundreds of similar requests, an agent might consolidate the pattern into a semantic rule: "customers who received damaged items within 7 days are eligible for express replacement." When a new request arrives, this semantic rule is loaded into working memory alongside the specific episodic details of the current case. If the agent also has stored scripts for processing returns (procedural memory), the procedure says how, semantic memory says what the policy is, episodic memory says what happened, and working memory holds the live reasoning context.

This four-layer integration is the aspiration; most current systems implement only two layers well and handle the transitions between layers via crude heuristics.

Why consolidation failures are silent - and why that matters

Orchestration failures in hierarchical memory tend to be silent. Unlike a crashed API call, which produces an error message, a paging decision that evicts the wrong record simply results in a slightly worse response - no exception, no log entry, no obvious signal that something went wrong.

This is the practical danger. A crashed API is noticed immediately. A memory that was quietly over-compressed, or evicted at the wrong moment, degrades the agent's answers by 5-10% across dozens of interactions before anyone traces the cause.

Most agent memory systems are explicit about retrieval architecture and silent about consolidation policy. That silence is where production failures live. The four levers any consolidation policy needs to specify are: which observations become memories at all (importance), how related facts get unified (merge), how confidence in old facts degrades (decay), and when memories leave the system entirely (eviction).

Omission is the dominant consolidation error type because memory construction is inherently compressive, while corruption and hallucination require actively writing incorrect or unsupported facts. This means the most common failure is not a hallucinated memory - it is a missing one.

How sandboxing connects to memory: the action layer

Memory tells the agent what it knows. Sandboxing controls what it can do with that knowledge. When an agent moves from retrieval to action - running code, writing files, calling APIs - those actions need an execution boundary.

Standard containers are not sufficient for AI-generated code because they share the host kernel. The three main isolation approaches are microVMs (Firecracker, Kata Containers), gVisor (user-space kernel), and hardened containers. MicroVMs provide the strongest isolation with dedicated kernels per workload, gVisor offers syscall interception without full VMs, and containers work only for trusted code.

The latency objection to microVMs - "a whole VM per request?" - is now largely answered. Firecracker microVMs deliver 100-125ms boot times, which is the current state-of-the-art for sandboxes. With snapshot-restore, you can get to ~28ms. AWS Lambda uses this for SnapStart: they freeze the VM after initialization, then restore the snapshot on cold start.

The deeper point: AI agents executing code at scale is turning sandboxing from a "developer tool" into an "infrastructure problem." The cost of choosing the wrong isolation approach is no longer a performance penalty - it is a security incident.

Isolation tier comparison

The tradeoff is real but smaller than it was two years ago. The right column is almost always the deciding factor, not the startup number.

For teams building agents in Slack or Teams, the memory stack and the sandbox tier are usually two separate infrastructure decisions - but they are architecturally linked. An agent that retrieves the wrong memory and then acts on it in a live environment causes the most damage precisely when the sandbox is weakest.

Agent memory types: common questions

What is working memory in an AI agent?

Working memory is the agent's current context window - everything loaded at inference time. It is fast and immediately accessible but bounded by the model's token limit and cleared at session end. Any fact not in the active context is invisible to the agent, which is why episodic and semantic stores exist.

What is the difference between episodic and semantic memory in an agent?

Episodic memory stores individual events with timestamps and context: "on Tuesday, the user asked about Q3 churn." Semantic memory stores distilled patterns extracted from many episodes: "this user always asks about churn at quarter-end." Episodic is raw; semantic is learned. Most systems implement episodic retrieval well but skip the consolidation step that builds semantic memory.

Why does agent memory consolidation fail silently?

Because a wrong paging or compression decision produces no error - just a slightly worse answer. Unlike a failed API call, which throws an exception, memory orchestration failures degrade outputs gradually, with no stack trace. This is why consolidation policy needs explicit design: importance scoring, merge rules, decay rates, and eviction criteria.

What is an agent memory sandbox and why does it matter?

Sandboxing is not strictly a memory concept - it is the execution boundary for actions the agent takes based on its memory. When an agent moves from retrieval to action (running code, writing to a database), isolation prevents a bad action from escaping to host infrastructure. MicroVMs like Firecracker give hardware-level isolation with boot times now under 30ms using snapshot-restore.

How do I choose between episodic and semantic retrieval for my agent?

Use episodic retrieval when the specific context of an event matters - who asked, when, what changed. Use semantic retrieval when the distilled rule matters more than the originating event. Most production agents need both, with a consolidation trigger that promotes repeated episodic patterns into semantic entries automatically rather than relying on a human to define all the rules upfront.