How RAG Works: From Your Question to the Right Document

A Fortune 500 company's internal chatbot once told employees their parental leave was 14 weeks. The policy had been updated 18 months earlier. Nobody caught it until an HR audit. The model wasn't broken - it was just answering from its training data, because nobody had built it a way to look anything up.

That's the problem Retrieval-Augmented Generation, RAG, solves. RAG is an AI framework that connects large language models to external knowledge sources at inference time. Instead of baking your company's facts into model weights - which is slow, expensive, and goes stale the moment someone updates a doc - RAG fetches the relevant text right before the model writes its reply.

Here is how that actually works, step by step.

The pipeline: five steps from question to answer

The core pipeline goes: chunk docs → embed → store in a vector database → retrieve on query → augment the prompt → generate. Each step is small and checkable, which is why RAG is easier to debug than fine-tuning.

1. Chunk. Your documents - PDFs, Notion pages, Confluence articles, support tickets - get split into short passages. Chunk size is measured in tokens; 512 tokens is roughly 2,000 characters, or about three to four paragraphs. The choice of size matters more than most people expect. Chunking configuration influences retrieval quality as much as - or more than - embedding model selection, according to Vectara's peer-reviewed NAACL 2025 study across 25 chunking configurations and 48 embedding models. That makes chunking the highest-leverage optimization most teams underinvest in.

Recursive 512-token splitting with 10-20% overlap is the benchmark-validated default for general RAG.

Factoid queries work well at 256-512 tokens, while analytical and multi-hop queries benefit from 512-1,024 tokens.

2. Embed. Each chunk gets passed through an embedding model, which converts the text into a list of numbers - a dense vector - that captures its meaning. Embeddings serve as the memory of the RAG system, where each vector encodes the meaning of a chunk. This allows the system to not just match keywords, but to retrieve relevant information based on context, even when queries use different phrasing.

3. Store. Those vectors go into a vector database - Pinecone, Qdrant, Weaviate, or ChromaDB are the four that dominate production. Qdrant delivers the lowest p50 latency at 6ms for 1 million vectors, running as a Rust-native binary with HNSW indexing.

Pinecone manages billions of vectors at heavy concurrency, often staying under 50 milliseconds for high-percentile latency.

4. Retrieve. When a user asks a question, the query gets embedded the same way the chunks were. The vector database finds the chunks whose vectors are closest in meaning. Here is where most production systems leave points on the table: they use dense semantic search alone. Sparse-only retrieval - BM25 keyword matching - scores 65% recall. Hybrid search combines both and hits 91% recall@10. That 17% improvement over dense-only comes from capturing keyword matches that embedding similarity misses.

The latency cost is minimal: hybrid search adds 6ms to the p50 versus dense-only, 18ms versus 12ms. At p99, the difference is under 15ms. No production system would reject a 17% recall improvement for 6ms of latency.

5. Generate. The top-K retrieved chunks get dropped into the prompt alongside the original question. At query time, code retrieves the most relevant parts of your information, stuffs it into the prompt, and asks the model to answer using that context. The model contributes language and reasoning; your data contributes facts.

This retrieval step grounds the output in current, verifiable evidence, which reduces hallucinations and improves factual accuracy.

RAG vs fine-tuning: where each one actually wins

RAG and fine-tuning answer different questions. Fine-tuning changes what the model is; RAG changes what it reads.

The pattern is: fine-tuning teaches how to respond; RAG provides what to respond with.

Fine-tuned models often generate smoother, more consistent outputs and perform better on repetitive domain-specific tasks. However, they may still produce outdated information if their training data is no longer current.

Fine-tuning looks operationally tidy until your knowledge base updates weekly and you find yourself retraining every two weeks to stay current - at which point RAG was always the right answer.

The combination beats either alone. Testing Llama 2-13B, GPT-3.5, and GPT-4 on a domain-specific QA task, fine-tuning alone improved accuracy by approximately 6 percentage points over baseline. RAG alone improved it by approximately 5 points. But the combination improved it by roughly 11 points - when two complementary systems each handle what they handle well.

The order matters: build RAG first, prove the use case, then fine-tune only the parts that retrieval cannot fix.

Where RAG quietly breaks in production

Understanding the happy path is easy. The failure modes are more useful.

Bad chunking kills good retrieval. If you split mid-sentence, or chunk too large, the vector for that passage represents five different ideas at once. A large chunk may encompass multiple topics, some of which are relevant to a user query while others are not. The representation of each topic within a single vector becomes diluted, which affects retrieval precision.

Embedding drift. Your embedding model and your chunks were aligned at index time. If you swap models later without reindexing, similarity scores become meaningless. Embedding drift over time quietly degrades the RAG side if you're not tracking retrieval recall separately.

Missing metadata and access controls. Enterprise RAG fails without governance: access controls, metadata, and context must precede retrieval. A retriever that surfaces confidential HR data to a contractor is not a retrieval bug - it's an architecture decision you made by skipping access filtering at the chunk level.

Skipping evals. The dominant RAG evaluation framework in 2026 is RAGAS, which decomposes quality into four metrics: faithfulness (does the answer use only the retrieved context?), answer relevancy (does the answer address the question?), context precision (are the right chunks being retrieved?), and context recall (is anything important being missed?). Teams that skip this end up debugging production failures by reading complaints in Slack.

When RAG shows up inside a Slack workflow

The same pipeline that powers a customer-facing bot can run inside a Slack channel. Someone asks a question in #hr-questions. An assistant queries the company's benefits docs, retrieves the right passage, and drafts a reply with a source link.

The draft-and-approve step matters here. RAG reduces hallucinations but does not eliminate them - a human reviewing the sourced answer before it posts is the last line of defence, and it costs about two seconds.

How RAG works: common questions

What does RAG stand for and what does it do?

RAG stands for Retrieval-Augmented Generation. It connects a language model to an external knowledge base at the moment of each query, so the model answers from your current documents rather than from its training data. This reduces hallucinations and keeps responses accurate when your information changes frequently.

Does RAG replace fine-tuning?

No - they address different problems. RAG updates what the model reads; fine-tuning changes how it reasons and responds. For most teams, RAG is the right first step because it requires no retraining. Fine-tuning makes sense only after RAG is in place and you've identified specific behaviors that retrieval alone cannot fix.

Why does chunk size matter in a RAG pipeline?

Chunk size determines how focused each retrieved passage is. A chunk that's too large dilutes the vector with multiple topics, reducing precision. A chunk that's too small loses surrounding context. The benchmark-validated default is 512 tokens with 10-20% overlap, which scored the highest composite accuracy across a 50-document real-world evaluation in early 2026.

What is hybrid search in RAG?

Hybrid search combines dense vector retrieval - semantic similarity - with sparse keyword retrieval like BM25. Dense search finds passages that mean the same thing even if they use different words. BM25 catches exact matches that vector search misses. Hybrid search hits 91% recall@10; the 17% improvement over dense-only comes from capturing keyword matches that embedding similarity misses.

Can RAG work inside Slack or Teams?

Yes. The same pipeline - embed your docs, store vectors, retrieve at query time - can run behind an assistant that lives in a channel. A question triggers retrieval, the top chunks get assembled into a prompt, and a draft answer surfaces for a human to approve before posting. The integration works on any knowledge source the assistant can read: Notion, Confluence, Google Drive, or a ticketing system.