How RAG Retrieval Actually Works, Step by Step

Most LLMs were trained on data with a cutoff date and have never seen your internal documents. When a model confidently answers a question about your Q3 roadmap or last week's incident, it is not remembering - it is reading. That reading mechanism is retrieval-augmented generation, and it is worth understanding precisely, because the failure modes are invisible until they are expensive.

What RAG actually does (and what it does not)

Retrieval-augmented generation is a technique for enhancing the accuracy and reliability of generative AI models with information fetched from specific and relevant data sources. The key word is "fetched." The model does not learn your documents. It reads a few relevant passages at inference time, the same way you might look something up before answering a question.

Fine-tuning is like a closed-book exam: the model internalizes domain knowledge during training and must rely solely on that at inference. While effective for learning formats or stylistic patterns, it is resource-intensive and inflexible when knowledge changes frequently. RAG is like an open-book exam: the model dynamically retrieves relevant information from an external source during inference, enabling it to access up-to-date and specialized information without retraining.

That distinction matters in practice. When information changes frequently - financial reports, policy docs, incident runbooks - RAG can retrieve the latest version without needing to retrain the model. Fine-tuning would require a new training run every time a document changes. RAG just requires re-indexing that document.

The three-stage pipeline: index, retrieve, generate

A typical RAG pipeline has three stages: indexing, retrieval, and generation. During indexing, a collection of external documents is preprocessed into a searchable knowledge base. The documents are divided into smaller text chunks, each capturing a focused piece of information while remaining short enough to fit within the LLM's context window. Each chunk is then encoded using an embedding model, which maps the textual content into a dense vector representation.

Stage 1 - Indexing (runs offline)

Before any query is answered, every document in your knowledge base gets cut into chunks and converted into vectors. Chunking is the process of splitting source documents into smaller units before embedding and indexing them for retrieval. Each chunk becomes a discrete retrieval unit. The chunking strategy determines how boundaries are drawn: by character count, sentence boundaries, document structure, semantic similarity, or LLM-proposed logic.

Common practice puts chunks between 128 and 512 tokens. Smaller chunks work well for fact-based queries where precise matching matters, while larger chunks are better for tasks requiring broader context, like summarizing concepts.

Poor chunking causes retrieval failures, hallucinations, and lost context in generation

• often before the LLM ever sees the query.

Once chunked, each piece gets embedded. Vector embeddings are numerical representations of text and other data types. They work by mapping complex, high-dimensional data into a lower-dimensional space using ML models, enabling computers to identify patterns and power semantic search. Two chunks that mean similar things will land close together in that space - even if they share no words.

Stage 2 - Retrieval (runs at query time)

When a user asks a question, that question is also converted into a vector using the same embedding model. The system then searches the index for the chunks whose vectors are closest to the query vector. This is the semantic search step: it finds relevant passages even when the user's exact words do not appear in the document.

Here is where pure vector search shows its limits. BM25 excels at exact-match queries - product codes, entity names, precise technical terms - but has no representation of semantic meaning. Vector search understands meaning but can miss specific identifiers. On the WANDS e-commerce benchmark, a tuned hybrid setup reaches 0.7497 NDCG - a 7.4% lift over either BM25 or pure vector search alone.

Production retrieval stacks combine both. A production retrieval stack is layered: BM25 plus vector search is fused with Reciprocal Rank Fusion (RRF) and optionally followed by a cross-encoder reranking stage for final relevance gains on a small candidate set.

RRF operates on ranks rather than raw scores, which solves the score-incompatibility problem - BM25 produces unbounded integers while cosine similarity is bounded - that makes naïve weighted averaging fail in production pipelines.

Stage 3 - Generation (the LLM's job)

Retrieved texts are prepended to the prompt - "Context: [retrieved info] ... Question: [user's question]" - so the LLM can see the extra context and use it to inform its response. Under the hood, the LLM is performing its usual next-token prediction, but because the prompt now includes relevant facts, the output will be "grounded" in those facts.

The failure mode nobody monitors: embedding drift

Embedding model generations move every 6-12 months and deliver 10-20 point lifts on retrieval benchmarks. A stack indexed on a 2024-era model and never re-embedded is one or two generations behind by 2026, with quality losses that match the model gap on real queries.

This is the non-obvious failure: your pipeline degrades without any errors or alerts. Embedding drift occurs when your query embedding model and document embedding model do not match exactly - and it happens more often than you might think. Teams update to newer embedding models for better quality, or switch providers for cost reasons, but forget to re-embed their entire document corpus.

Embedding model upgrades create a specific failure mode that is easy to miss in monitoring: index drift. Every embedding model maps text into a high-dimensional space with its own geometry - directions, distances, and neighborhood structures.

When you change the model, the coordinate system changes. A vector from an old model and a vector from a new model are not comparable - so your new queries and your old chunks no longer speak the same language.

There is a parallel failure with domain-specific vocabulary. An insurance company using terms like "pool" (a risk-sharing instrument, not a swimming pool), "Solvency II Article 7," or proprietary product tier names faces a problem: none of this is in the model's training distribution. On enterprise terms, embeddings fail the same way older models fail on ambiguous words, because the institutional sense the embedding would need to recover is not institutionalized anywhere outside that company.

The fix is not a bigger embedding model. It is a keyword layer - BM25 - running in parallel, catching the exact-match cases that semantic search cannot.

When to reach for RAG vs. fine-tuning

As a general rule: choose RAG when you need recent, up-to-date knowledge; fine-tuning when you need specific behavior or tone; and a hybrid approach when your project requires both.

How RAG retrieval works: common questions

What is retrieval-augmented generation in simple terms?

RAG lets an LLM read from your documents at the moment a question is asked, rather than relying only on what it learned during training. The system converts your documents into searchable vectors, finds the most relevant passages at query time, and feeds those passages to the model as context before it generates its answer.

What is the difference between RAG and fine-tuning?

Fine-tuning rewrites the model's internal knowledge. RAG adds an index so you can look things up. Fine-tuning is better for stable knowledge and specific output formats. RAG is better when documents change frequently and when you need traceable, source-attributed answers.

What chunk size should I use for RAG?

Common practice puts chunks between 128 and 512 tokens. Smaller chunks work well for fact-based queries where precise matching matters; larger chunks are better for tasks requiring broader context. Start around 400 tokens with 10-15% overlap and adjust based on your actual query distribution.

Why does my RAG system get worse over time without any changes?

This is likely embedding drift. Most retrieval problems blamed on the model are not model problems. They are embedding drift problems caused by nondeterministic preprocessing, inconsistent chunk boundaries, partial re-embeddings, or drifting indexes - and once embeddings drift, the model cannot save the pipeline. Re-embed your corpus whenever you change or upgrade your embedding model.

Should I use pure vector search or hybrid search for RAG?

Hybrid search. If your RAG system uses pure vector search, adding BM25 is the single highest-impact retrieval upgrade you can make. BM25 catches exact-match queries that vector search misses - product codes, names, version numbers - while vector search handles paraphrased or conceptual queries. Fuse them with RRF and you cover both cases.