How RAG Works: From Your Question to a Grounded Answer

Ask a plain LLM what your company's parental-leave policy says, and it will confidently invent something. RAG - retrieval-augmented generation - is the architecture most teams use to stop that. RAG reduces hallucinations by 71% on average compared to non-RAG systems. But despite how often the term gets thrown around, the actual mechanics stay fuzzy. This is what happens inside a RAG pipeline, from the moment you hit send to the moment a grounded answer comes back.

The two phases: indexing and retrieval

A complete RAG pipeline runs in two phases: an ingestion phase (documents → chunking → embedding → vector database) and a retrieval phase (user query → embed query → similarity search → top-K results → LLM generation → response).

The first phase - indexing - happens before any user ever asks a question. The second - retrieval - happens in real time.

Indexing, step by step:

1. Chunk your documents. Chunk size is typically tuned to balance context completeness and specificity - chunks must be large enough to contain useful context, yet small enough to match queries narrowly and fit within model context windows. A 300-word policy document might become three or four chunks; a 50-page handbook might become hundreds.

2. Embed each chunk. Each chunk is embedded into a high-dimensional vector representation that encodes its semantic content, usually done with a transformer-based bi-encoder that produces dense vector embeddings of text. OpenAI's text-embedding-3-small, for instance, costs $0.02 per million tokens and transforms text into 1,536-dimensional vectors for semantic search and RAG applications. Embedding 10,000 documents of 500 tokens each - five million tokens total - costs about ten cents.

3. Store in a vector database. Text chunks are embedded into a vector space where similar chunks are close to each other, allowing fast nearest-neighbor search. Pinecone, Weaviate, Qdrant, and Milvus all handle this; they differ mainly in hosting model and pricing, not in the underlying idea.

Retrieval, step by step:

1. Embed the question. When you type a question, the same embedding model converts it to a vector.

2. Run a similarity search. The vector database finds the stored chunks whose vectors are closest to the question vector, usually measured as cosine similarity. The top-K chunks are returned - commonly K=3 to K=10.

3. Stuff the context into the prompt. The retrieved texts are prepended to the prompt - "Context: [retrieved info]... Question: [user's question]..." - so the LLM can see the extra context and use it to inform its response.

4. Generate. The LLM performs its usual next-token prediction, but because the prompt now includes relevant facts, the output is "grounded" in those facts.

Why the retrieval step is where things actually break

Most people assume the LLM is the fragile part. The numbers say otherwise. Industry analysis in 2026 consistently shows that when RAG fails, the failure point is retrieval 73% of the time, not generation.

There are two root causes worth understanding.

Chunking breaks meaning. Fixed-size chunking - splitting every 512 or 1024 characters regardless of content - cuts sentences mid-thought, separates questions from their answers, and drops the context that makes a passage meaningful. A 2025 clinical study cited by production RAG practitioners found adaptive chunking achieved 87% retrieval accuracy versus 13% for fixed-size baselines on the same dataset

• an extraordinary gap, and one most teams running the default LangChain chunker never measure.

Embeddings are lossy. Bi-encoder vector embeddings are "lossy" by design - they compress a complex paragraph into a single point in a 1,536-dimensional space. A user asking to compare Q3 2025 revenue to Q3 2024 might get the 2024 data back, because the semantic distance between "2024" and "2025" is negligible to an embedding model. The model retrieved something that looked right but was one year off.

The non-obvious consequence: more context is not the fix. Giving an LLM more information can make it dumber. A 2025 study by Chroma tested 18 powerful language models - including GPT-4.1, Claude, and Gemini - and found every single one performed worse as input grew. Some models held at 95% accuracy and then nosedived to 60% once input crossed a certain length. Dumping all retrieved chunks into the context is not a safety net; it's a way to confuse the model with noise.

Hybrid search and reranking: how production systems close the gap

Vector search retrieves semantically similar passages and handles paraphrase and concept matching well, but misses exact keyword matches. BM25 handles exact matches and rare terms well but misses semantic relationships. Neither alone is sufficient for a production RAG system that handles varied query types.

The practical solution is hybrid retrieval - run both, merge the ranked lists using Reciprocal Rank Fusion, then rerank the combined results with a cross-encoder (a model that scores query-document pairs jointly rather than independently). When both hybrid retrieval and contextual techniques are combined, error rates drop by roughly 69% compared to naive vector-only retrieval.

Semantic chunking improves recall up to 9% over fixed-size approaches

• enough to matter when you have thousands of users. Voyage-3-large outperforms OpenAI and Cohere embeddings by 9-20% on retrieval benchmarks, which is worth knowing if you're optimizing past the defaults.

Where a RAG pipeline fits inside a Slack-connected AI workflow

RAG is infrastructure, not a product. The pipeline lives behind whatever AI assistant your team interacts with. When someone in Slack asks about a policy, a budget, or an engineering spec, a well-built RAG layer finds the right chunk from the right document and hands it to the model - so the model answers from your content, not from its training.

The hard part isn't the API calls. It's the decisions made in the first week: chunk size, overlap percentage, embedding model, whether to use hybrid search, how often to re-index when documents change. If you need to re-index documents, multiply the embedding cost by frequency - monthly updates mean 12× the annual indexing cost. That math is easy to overlook at prototype stage and expensive to fix later.

A teammate like Beagle lives inside that retrieval loop at the point where the answer surfaces in Slack - drafting the reply, citing the source, and waiting for a human to approve before anything posts. The retrieval does the finding; the human-in-the-loop does the trusting. Both matter.

How RAG works: common questions

What is the difference between RAG and fine-tuning?

Fine-tuning bakes knowledge into the model's weights during training. RAG keeps knowledge outside the model and retrieves it at query time. RAG is cheaper to update - change a document, re-index it, done - while fine-tuning requires a full training run. For frequently changing or proprietary information, RAG is almost always the right starting point.

How many chunks does a RAG system retrieve per query?

Most production systems retrieve 3-10 chunks (K=3 to K=10) per query, then pass those chunks as context alongside the user's question. Returning more chunks adds noise and can degrade the model's answer quality; some models hold at 95% accuracy and nosedive to 60% once input crosses a certain length. More is not safer.

Why does RAG still hallucinate sometimes?

Hallucination persists even with correct retrieval when noise, contradictions, or strong training priors override retrieved evidence. The model can surface the right chunk and still ignore it. Reranking, explicit citation requirements in the prompt, and evaluation pipelines that check whether the answer is grounded in the retrieved text all reduce this - but none eliminate it entirely.

What chunk size should I use?

There is no universal answer. Shorter chunks (256-512 tokens) often produce better retrieval results than embedding entire documents , because a small chunk matches a narrow query more precisely. A good default is 300-500 tokens with 10-20% overlap between adjacent chunks, then test recall@K against your actual query types before locking anything in.

How do I know if my retrieval is actually working?

Track Precision@K - the fraction of your top-K retrieved chunks that are actually relevant to the query - and Recall@K, the fraction of all relevant chunks you successfully surfaced. Mean Time to Answer should stay under three seconds for interactive use cases, and Answer Rate - the percentage of queries that produce usable responses - should target above 90%. Log misses; they will tell you where your chunking strategy breaks before your users do.