Ask most LLMs a question about your company's internal pricing tiers and they will either hallucinate an answer or admit they have no idea. That gap - between what a model was trained on and what it actually needs to know - is the problem retrieval-augmented generation exists to solve. It does not retrain the model. It fetches the relevant text at the moment someone asks, stuffs it into the prompt as context, and lets the model reason over real content rather than guessing.
This is a surprisingly mechanical pipeline once you see all the moving parts. Let's walk through it end to end, using a concrete example: a support team that wants an LLM to answer questions about a 400-page technical manual that gets updated quarterly.
What chunking actually does - and why it is the highest-leverage decision
A typical RAG system has three main components: a retriever, a language model generator, and an extensive knowledge database. But before any of that can work, every document in that knowledge base has to be split into smaller pieces. This step is called chunking, and if you only fix one thing in a struggling RAG pipeline, fix chunking. Chunking defines the units of knowledge your system can retrieve.
For the technical manual example: you would not embed all 400 pages as one giant blob - no similarity search could sensibly match against something that large. Instead you split it. A common production baseline is 512 to 1,024 tokens per chunk, with roughly 20-25% overlap between adjacent chunks to reduce the risk of splitting key sentences or step sequences. Avoid splitting in the middle of headings, lists, or code blocks.
The overlap detail matters more than it sounds. A chunk boundary failure happens when the answer exists in your documents but got split across two chunks. Neither chunk alone is relevant enough to retrieve, so the user gets a partial answer or no answer at all.
There is also a deeper structural tension in the pipeline. The traditional "chunk-embed-retrieve" approach asks a single-granularity, fixed-size text chunk to serve two conflicting tasks: semantic matching (recall), which favors smaller chunks of 100-256 tokens for precise similarity search, and context understanding (utilization), which needs larger chunks of 1,024+ tokens for coherent generation. This creates a difficult trade-off between "precise but fragmented" and "complete but vague."
One way teams resolve this is semantic chunking - splitting where meaning actually shifts rather than at a fixed token count. Compute embeddings sentence-by-sentence and start a new chunk when cosine similarity between adjacent sentences drops below a threshold. One published comparison reported semantic chunking lifting accuracy to around 71% versus fixed-size baselines on the same dataset.
How vector embeddings and similarity search find the right chunks
Once you have chunks, you embed them. An embedding model converts each chunk of text into a high-dimensional numerical vector - a list of floating-point numbers, typically between 384 and 1,536 dimensions - that encodes the chunk's meaning. When a user asks the LLM a question, that query is also sent to a model that converts it into a numeric format. The numeric version of the query is sometimes called an embedding or a vector.
The embedding model then compares these numeric values to vectors in a machine-readable index of the available knowledge base. When it finds a match or multiple matches, it retrieves the related data and passes it back to the LLM.
The intuition: vector search retrieves based on semantic similarity, where "similar" means "close in embedding space." This means it can match "terminate the contract" with "cancel the agreement" even when the words barely overlap. For our technical manual, a user asking "what's the voltage limit on the main bus?" can surface a chunk that talks about "maximum bus voltage threshold" - same concept, no shared keywords.
The same embedding model family and configuration must be used for both indexing and querying, or retrieval quality will degrade. This is an easy mistake to make when you swap models mid-project.
Why most teams need hybrid search and a reranker
Pure vector search has a specific failure mode. Semantic-but-not-exact misses happen when a user searches for "PostgreSQL 17 performance improvements" and vector search returns generic database performance content because it understands the semantics of "performance" but doesn't weight the exact version number.
Hybrid search addresses this: you use keyword retrieval when users search for identifiers, names, part numbers, or exact phrases, while still capturing semantic matches with embeddings. In practice this usually means running a dense vector search and a BM25 keyword search in parallel, then fusing the ranked lists - a technique called Reciprocal Rank Fusion. A mature pipeline looks like: query → optional query rewrite → hybrid retrieval (dense top-50 plus BM25 top-50, RRF fused) → reranker → top 5-8 results → context builder → LLM.
The reranker is what cleans up the fusion results. It is a separate model (Cohere Rerank and Voyage AI are common choices) that takes the candidate chunks and re-scores them with a more expensive cross-attention pass - looking at the query and each candidate together, not separately. Production systems typically retrieve top-K chunks, where K is often 5-10. Retrieve 20, rerank to the best 5, and you get substantially fewer irrelevant passages polluting the context window.
Retrieving 10 chunks when only 2 are relevant dilutes the signal. The LLM averages across all context, producing a mediocre answer. This is one of the most common reasons teams see fluent but vaguely wrong responses - the model is doing its best to reconcile several conflicting or irrelevant passages.
The augmentation step: how retrieved text reaches the model
Once relevant chunks are retrieved, they are combined with the user query into a prompt that provides the model context for the generation step. This is the "augmentation" in retrieval-augmented generation. For our technical manual, it looks something like: