A support engineer asks their internal AI tool: "What's the SLA for enterprise tier?" The model answers correctly, with the right number, citing the correct contract doc. It didn't memorize that number during training - it looked it up, five seconds ago, before composing its reply.
That lookup is retrieval-augmented generation. It sounds like a technique. It's really closer to giving a model a working memory it doesn't have by default. Understanding how that works - chunk by chunk, vector by vector - changes how you build with AI and what you trust it to get right.
What RAG actually does (and what it doesn't)
Instead of relying solely on static training data, RAG retrieves relevant documents at query time and feeds them into the model as context. That one sentence contains most of what you need to understand. The model's weights stay frozen. What changes is what gets stuffed into the prompt right before it generates an answer.
The classic analogy holds: it is an open-book exam. A student taking a closed-book exam relies on what they have memorized - that's a standard LLM. It's smart and can reason, but limited to what's in its parameters. A student with an open-book exam has the same reasoning skills but can check relevant pages before answering. That's RAG.
What RAG doesn't do: it doesn't make the model smarter, it doesn't update its beliefs, and it doesn't give it access to everything in your docs at once. A single call with nearly 1M tokens can be orders of magnitude more expensive than RAG's lean prompts, and as context grows, relevant passages may be "buried" - LLMs can struggle to maintain attention over very long, heterogeneous content. RAG's whole job is to avoid that problem by finding the right passages before the model ever sees them.
The four steps of a RAG pipeline
RAG pipelines typically involve four steps: document preparation and chunking, vector indexing, retrieval, and prompt augmentation. Here's what each one actually does.
1. Chunking
You can't embed a 300-page PDF as one thing. Handle long documents by chunking them into smaller sections (usually 500-2000 tokens), storing chunks in a vector database with embeddings, and retrieving only relevant chunks. Getting chunk size wrong costs you downstream. Recursive 512-token splitting with 10-20% overlap is the benchmark-validated default for general RAG. But it's worth knowing that chunking configuration influences retrieval quality as much as embedding model choice.
One underrated move: metadata enrichment on chunks boosts QA accuracy from ~50-60% to 72-75% without changing retrieval architecture. Prepend a title, section header, and document date to each chunk before embedding it. That extra context shapes how the embedding model represents the text.
2. Embedding
The system does not understand actual English words in the chunks, so they get converted into numerical representations. Text is converted into numerical vectors called embeddings. These vectors capture the semantic meaning of the text.
Each chunk becomes a high-dimensional vector - often 768 or 1536 floating-point numbers - that encodes what the text means, not what words it contains. Two sentences that say the same thing in different words land close to each other in vector space. Two sentences that use similar words but mean different things land far apart.
A 5% improvement in embedding quality can mean the difference between a RAG system that answers correctly and one that hallucinates. The embedding model is the one component that most teams underspecify. Most comparisons rank embedding models by overall MTEB scores. That average blends classification, clustering, summarization, and retrieval tasks into one number. For RAG, only the retrieval subset matters.
3. Vector retrieval
At query time, the user's question gets run through the same embedding model. That query vector gets compared against every stored chunk vector to find the closest matches.
The predominant architecture for scalable dense retrieval is the bi-encoder, which independently maps queries and documents into a shared vector space and estimates their relevance via cosine similarity. This design enables offline indexing of the document corpus, after which subsequent queries can be resolved in near-constant time using approximate nearest neighbor (ANN) search, allowing the method to scale to hundreds of millions of passages in production environments.
"Approximate" is doing work there. Approximate Nearest Neighbors (ANN) is a class of algorithms used to find the nearest neighbors of a query point in a high-dimensional dataset. These algorithms are called "approximate" because they trade off a small amount of accuracy for a significant speedup compared to exact nearest neighbor search algorithms. In practice: retrieval from a million-chunk corpus happens in under 50ms. Exact search over the same corpus would be unusably slow.
4. Reranking
This is the step most explainers skip, and it's the one that separates demo-quality RAG from production RAG.
A cross-encoder reranker is a second retrieval pass that takes a query and a candidate passage, runs them through a transformer with full attention across both, and returns a single relevance score. Unlike the bi-encoder in Stage 1, a cross-encoder actually reads the query and the passage together - it can detect things like "this passage mentions the word but answers a different question."
You let the cheap retriever cast a wide net (k=100 or k=200), then run a slower, smarter model over those candidates to push the actually-relevant ones to the top. The LLM only sees the top 5 or 10 after rerank.
Drop one between your vector store and your LLM, and a typical production RAG pipeline sees NDCG@10 lift commonly in the 5-15 point range, and 20+ on lexically hard datasets, for under 200ms of added latency.
Where things go wrong in practice
The pipeline looks clean on a diagram. Production is messier.
Retrieval returns the right document, wrong passage. If a contract's SLA clause is on page 47 but your chunks follow fixed-size splits, the clause might be split across two chunks and neither retrieves cleanly. Overlap helps; section-aware chunking helps more.
The embedding model hasn't seen your domain. Embedding model MTEB scores don't predict domain-specific performance; always benchmark on your own data before committing. A model that scores well on academic text may rank code documentation badly.
Too many chunks reach the LLM. The "Lost in the Middle" finding showed that solely the location of the "golden document" within a long context significantly affects the performance of language models. Passing 20 retrieved chunks to a model isn't neutral - the model will underweight passages buried in the middle of a long context. Pass fewer, better ones.
Stale chunks. To maintain current information for retrieval, asynchronously update the documents and update embedding representations of the documents. You can do this through automated real-time processes or periodic batch processing. Teams that skip this build systems that confidently answer with outdated policy.
The moment it reaches the model
After chunking, embedding, retrieval, and reranking, the LLM finally gets involved. It sees something like: