Your company's internal knowledge base has 40,000 documents. An LLM trained six months ago knows none of them. When someone asks your AI assistant a question about last quarter's pricing, the model doesn't dig through those docs the way you'd picture a researcher digging through a filing cabinet. It runs a math problem against a database of floating-point numbers-usually in under 100 milliseconds. That process is RAG: Retrieval-Augmented Generation. Here's what actually happens.
What RAG actually does, step by step
RAG is a technique that enables large language models to retrieve and incorporate new information from external data sources. That one sentence hides four distinct mechanical steps, each with real trade-offs.
Step 1: Chunking. Your documents don't go into the system as whole files. They get cut into pieces first. Chunking is the segmentation of documents into retrievable units prior to embedding and indexing-it governs how semantic information is represented in vector space and how faithfully retrieved evidence reflects the source text. The chunk size decision matters more than most teams realize. A common approach splits documents using a recursive character splitter at 512 tokens with an overlap of 50 tokens, using a priority-ordered separator hierarchy: paragraph breaks, line breaks, sentence boundaries, then word boundaries.
The 10% overlap mitigates boundary artifacts where a relevant sentence straddles two consecutive chunks. Go too small and you lose meaning: at a granularity of 8 tokens, both fixed and semantic chunking strategies produce near-zero nDCG scores, because individual segments are too short to preserve coherent semantic content.
Step 2: Embedding. Each chunk gets converted into a vector-a list of floating-point numbers that encodes its meaning. When a user asks a question, the AI model sends the query to another model that converts it into a numeric format so machines can read it-this numeric version of the query is called an embedding or a vector.
The predominant architecture for scalable dense retrieval is the bi-encoder, which independently maps queries and documents into a shared vector space and estimates their relevance via cosine similarity-this design enables offline indexing of the document corpus, after which subsequent queries can be resolved in near-constant time using approximate nearest neighbor (ANN) search.
Step 3: Vector search. At query time, the user's question gets embedded using the same model, and the system finds the chunks with the closest vectors. The embedding model compares numeric values to vectors in a machine-readable index, and when it finds a match or multiple matches, it retrieves the related data, converts it to human-readable words, and passes it back to the LLM.
Step 4: Generation. The retrieved chunks get stuffed into the LLM's context window alongside the original question. The model generates an answer grounded in that retrieved text instead of relying purely on what it learned during training.
Why the first retrieval pass isn't enough
This is the step most explanations skip, and it's where most RAG systems leak quality.
The bi-encoder that handles vector search is fast but imprecise. Bi-encoders must compress all of the possible meanings of a document into a single vector-meaning information is lost-and they have no context on the query because embeddings are created before any user query arrives.
In production-grade RAG applications, bi-encoders typically achieve only 65-80% relevance accuracy on complex queries, meaning 20-35% of the information fed to the LLM is noisy or irrelevant.
The fix is a reranker-specifically a cross-encoder. A cross-encoder reranker takes a query and a candidate passage, runs them through a transformer with full attention across both, and returns a single relevance score.
While a bi-encoder maps the query and document independently, the cross-encoder allows every token in the query to "attend" to every token in the document, enabling the model to learn causal relationships that might not be captured in a static vector embedding.
The standard pattern: retrieve 100 candidates with a bi-encoder, rerank to top 10; for latency-sensitive applications, retrieve 50 and rerank to top 5.
| Stage | Architecture | Latency | Accuracy | Scales to |
|---|---|---|---|---|
| Vector search | Bi-encoder | ~15ms / 1M docs | 65-80% relevance | Billions of docs |
| Reranking | Cross-encoder | 50-150ms / 20 docs | 85-90% relevance | Dozens of candidates |
Drop a reranker between your vector store and your LLM, and a typical production RAG pipeline sees NDCG@10 lift commonly in the 5-15 point range, and 20+ on lexically hard datasets, for under 200ms of added latency-that is not a marginal improvement; that is the difference between a RAG system that hallucinates and one that cites.
Beagle in action#product-team, 2:47pmThe concrete cost of each component
RAG isn't free infrastructure. Each layer has a bill.
Embedding cost: OpenAI's text-embedding-3-small runs at $0.02 per million tokens; text-embedding-3-large costs $0.13 per million tokens and produces 3072-dimensional vectors.
At $0.13/M, text-embedding-3-large is 6.5x more expensive than small-and the MTEB Retrieval benchmark advantage is roughly 2-3 percentage points. For most applications, that's not worth the price difference.
A practical number:
a 500-word chunk is approximately 375-400 tokens; add 10-25% for chunk overlap in RAG pipelines.
A corpus of 100,000 standard-length documents at 500 words each costs roughly $2-3 to embed with text-embedding-3-small. The surprise isn't the embedding bill-it's the storage.
Generating 1 billion embeddings with OpenAI small costs $20 once. Storing those vectors in a managed vector database can cost $200+ per month, indefinitely.
Latency by embedding provider: Running your own open-source embedding model changes the equation. Google's GE2 API model shows a 231.6ms median query latency with a p95 of 575.5ms, while self-hosted open models like BGE-M3 and E5-large run at roughly 30-31ms median.
The high variance in hosted API models is attributable to network round-trip time and autoscaling cold starts rather than model inference time.
Self-hosted BGE-M3 on a spot A100 GPU costs roughly $0.001 per million tokens-the cheapest option overall, but it requires infrastructure management.
What RAG still gets wrong
RAG reduces hallucinations, but it doesn't eliminate them. One controlled study found RAG eliminated hallucinations from 8% to 0% in 100 synthetic medical consultations under clean conditions. But real corpora are messier. Empirical studies show that retrieval-augmented legal research tools exhibited hallucination rates up to 33%, contradicting vendor claims.
This occurs when retrieved passages are topically relevant but factually insufficient, when multiple documents contain conflicting information, or when the model succumbs to the "lost-in-the-middle" effect.
The non-obvious insight: retrieval quality and generation quality fail independently. Hallucination in RAG arises from systematic retrieval-generation misalignment rather than retrieval failure alone-"Evidence Override" dominates, at 28.4% of hallucinations in medical datasets and 42.3% in HotpotQA, exceeding simple retrieval failure by 4-7x. The primary reason is generation-side grounding: relevant evidence is often successfully retrieved but ignored during generation.
That's the part a better embedding model won't fix. It requires prompting the model to actually cite and follow the retrieved context, or using techniques like Corrective RAG that verify whether retrieved chunks are being used.
A teammate like Beagle-connected to a team's Slack channels and linked documents-runs the same retrieval loop every time it assembles a draft: pull the relevant source, surface the evidence, let a human check before anything posts.
RAG retrieval: common questions
What is RAG and how does it work?
RAG (Retrieval-Augmented Generation) is a technique that gives an LLM access to external documents at query time. The query gets converted into a vector, a database finds the closest matching chunks, and those chunks are placed into the model's context window alongside the question. The model answers using that retrieved evidence instead of relying solely on its training data.
What chunk size should I use for RAG?
For most text corpora, 256-512 tokens per chunk with 10-20% overlap is the practical starting point. Shorter chunks lose semantic coherence; research shows retrieval quality collapses below 32 tokens. For long-form documents like legal filings or technical manuals, saturation may occur closer to 128-256 tokens. Measure recall on your specific corpus before committing.
What is a reranker and why does RAG need one?
A reranker is a cross-encoder model that scores each query-document pair jointly, with full attention across both texts. Standard vector search uses bi-encoders that embed documents independently, achieving 65-80% relevance accuracy. Adding a reranker after the first retrieval pass raises that to 85-90%-typically with under 200ms of added latency-by rescoring the top candidates before passing them to the LLM.
How much does it cost to run a RAG pipeline?
Embedding a typical 100,000-document corpus costs roughly $2-3 with OpenAI's small model at $0.02/M tokens. The ongoing cost is vector database storage, which can reach $200+/month at scale. Query latency costs are minimal at low volume but compound quickly: at 10,000 daily queries, embedding API calls alone can exceed the storage cost if you're using a hosted provider with high-latency APIs.
Does RAG actually prevent hallucinations?
It reduces them significantly in controlled settings-one medical study saw hallucinations drop from 8% to zero with clean, domain-specific sources. In production, with messy corpora or conflicting documents, hallucination rates can remain substantial. The leading failure mode isn't bad retrieval: it's the LLM ignoring retrieved evidence during generation, a problem that requires prompt-level and architectural fixes, not just better vector search.