Your teammate asks the agent, "What did we decide about the data retention policy?" The agent was not present for that meeting. It has no persistent brain. But within a second, it surfaces the right answer from a Notion page written in January. What just happened under the hood is worth understanding, because the same mechanism powers almost every "smart" AI feature your team encounters.
This is retrieval-augmented generation, almost always shortened to RAG - and it is the dominant way production AI agents handle memory today.
What RAG actually does
The short answer: instead of cramming everything into the context window, the agent stores knowledge in a searchable database, and when it needs information, it searches for relevant chunks and pulls only what it needs.
The longer answer involves a conversion step most people never see. Before any retrieval happens, every piece of knowledge in your corpus - meeting notes, Confluence pages, Slack threads you've indexed - gets turned into a vector.
The embedding model converts text into high-dimensional vectors that capture semantic meaning, forming the foundation for semantic search and RAG pipelines.
OpenAI's text-embedding-3-small,
released on January 25, 2024, produces embeddings with lower costs, higher multilingual performance, and a new parameter for shortening embeddings.
By default, the embedding vector is 1,536 numbers long for text-embedding-3-small.
Those 1,536 numbers are the document's address in a high-dimensional space where meaning determines proximity.
When the agent receives your query - "What did we decide about the data retention policy?" - it embeds that query using the same model, producing its own 1,536-number vector. Then it searches the database for vectors that sit nearby. It converts the current query into a vector and performs a similarity search, often using cosine similarity, to find the nearest vectors in that high-dimensional space. The closest matches come back as retrieved chunks, and those chunks get dropped into the model's context window alongside your question. The model then generates an answer grounded in what it retrieved rather than what it was trained on.
Vector search is based on similarity, not true understanding - the most similar stored embedding may not always be the most relevant in context. That gap is real. Two documents can score similarly to a query but mean very different things, which is why good production systems add metadata filtering, reranking steps, and sometimes a second model to score retrieved chunks before passing them forward.
How documents get sliced up before indexing
Before a document can be retrieved, it has to be chunked - cut into pieces small enough to embed precisely. This is unglamorous work, but it controls retrieval quality more than almost any other choice.
Chunking solves a retrieval precision problem: a 200-token chunk about "Python async/await" will rank higher for that query than a 5,000-token chapter about "Python concurrency." But good chunking also maintains semantic boundaries, because breaking mid-sentence destroys meaning.
The practical baseline that holds up in most settings:
start with RecursiveCharacterTextSplitter at 400-512 tokens with 10-20% overlap.
The overlap matters because critical context often straddles a chunk boundary - without it, a sentence that starts near the end of chunk 4 and finishes at the start of chunk 5 never appears whole.
A January 2026 systematic analysis identified a "context cliff" around 2,500 tokens where response quality drops, and separately found that sentence chunking matched semantic chunking up to ~5,000 tokens at a fraction of the cost.
The three kinds of memory an agent actually needs
RAG handles one category of memory - factual knowledge stored and retrieved on demand. But a well-built agent needs two others, and conflating them is a common architectural mistake.
Episodic memory captures specific past experiences with temporal details, best stored using vector databases for semantic search and event logs for ground truth. Semantic memory stores factual knowledge independent of specific experiences: customer profiles, product specs, domain expertise.
The third type, procedural memory, is different in kind. It stores learned skills and operational knowledge - for example, an agent could learn the optimal process for booking flights, like ensuring appropriate layover times or avoiding specific airports based on past connection failures. Procedural memory doesn't live in a vector store; it lives in how the agent is prompted and what workflows it has been trained or instructed to follow.
Most teams building on top of a foundation model are unknowingly using all three. The system prompt carries procedural memory. The context window carries short-term or episodic memory from the current session. RAG carries the semantic long-term store. When an agent gives a confidently wrong answer, the failure usually traces back to one of those three stores having stale, missing, or conflicting content - not to the model being incapable.
Where retrieval breaks - and what to do about it
The architecture described above has a specific failure mode that shows up constantly in production: models are constrained by static training data and limited context windows, which creates two challenges - a knowledge cutoff (no access to real-time or proprietary data), and memory limitations (no recollection of past interactions across sessions). RAG solves the first; it does not automatically solve the second.
If a user has a five-session thread of back-and-forth with an agent about a project plan, and nothing from those sessions is ever written to the episodic store, the agent starts each new session completely cold. An AI-powered support agent, for example, can remember previous interactions with a user and tailor responses accordingly - but only if that memory was explicitly captured and stored. The "remember" part is never automatic.
A practical pattern for handling this: at the close of each session, have the agent write a short structured summary - a date-stamped record of what was decided, what tasks are open, and what context matters next time. That summary gets embedded and stored. Next session, the agent retrieves it. Including metadata like timestamps, topics, or user intent in the retrieval process helps narrow down results to what is truly relevant at the moment.
An AI teammate like Beagle, living inside Slack or Teams, applies exactly this pattern: indexing conversation summaries so that follow-up questions days later can reach back to the thread that started them. The retrieval plumbing is invisible to the user. What they experience is an agent that does not make them repeat themselves.
The system is not magic. It is text converted to numbers, numbers compared by geometry, and the closest matches handed to a language model. Understanding that chain does not make it less useful - it tells you exactly where to look when it fails.