Your engineering wiki has 400 pages. A teammate asks the AI assistant: "What's the on-call escalation path for the payments service?" Three seconds later, it answers correctly, with a source link. It didn't memorize that page. It found it, mid-conversation, using a pipeline called retrieval-augmented generation - and the mechanics are worth understanding, because they determine exactly when it works and when it fails.
What "retrieval-augmented generation" actually means
RAG is a two-step architecture: look something up, then write an answer using what you found. RAG is a technique where an LLM queries an index - like a search engine, knowledge base, or vector database - to find additional, contextually relevant information for its response, rather than just defaulting to what it learned during training. The "augmented" part is literal: retrieved text gets inserted into the model's input, extending what it knows for that one call.
This matters because LLMs' reliance on static training data limits their ability to respond to dynamic, real-time queries, resulting in outdated or inaccurate outputs. A model trained last year doesn't know about the refactor your team shipped in March. RAG closes that gap without retraining anything.
Step one: chopping your docs into searchable pieces
Before any query lands, there's an offline indexing phase. Every document - wiki pages, runbooks, Notion docs, GitHub READMEs - gets split into chunks and converted into vectors.
Chunking is where most teams underestimate the complexity. The best chunk size for most RAG pipelines is 300-500 tokens. This balances retrieval precision - finding relevant chunks - with context preservation - keeping enough surrounding text for the LLM to understand.
The tension is real: using a single-granularity, fixed-size text chunk to perform two inherently conflicting tasks - semantic matching (recall), where smaller chunks of 100-256 tokens are needed for precise similarity search, and context understanding (utilization), where larger chunks of 1024+ tokens are needed for coherent LLM generation - creates a difficult trade-off between "precise but fragmented" and "complete but vague."
Production teams often land on a middle path. 512 to 1024 tokens per chunk is a common production baseline, with 20-25% overlap between adjacent chunks to reduce the risk of splitting key sentences, definitions, or step sequences.
Embedding converts each chunk into a vector - a list of numbers that encodes its meaning. Embeddings are numerical representations that capture semantic relationships between text. A piece of text - whether a word, phrase, sentence, or paragraph - is converted to a vector of numbers, typically 384-1536 dimensions. Similar meanings land close together in that high-dimensional space; unrelated meanings land far apart.
| Chunk size | Retrieval precision | Context quality | Typical use case |
|---|---|---|---|
| 100-200 tokens | High | Low - loses surrounding context | Keyword-heavy Q&A |
| 300-500 tokens | Good | Good | General knowledge bases |
| 512-1024 tokens | Medium | High | Long-form docs, policies |
| 1000+ tokens | Low - embedding averages out | Excellent | Rarely optimal |
Once chunks are embedded, the vectors go into a vector database - Pinecone, Weaviate, pgvector, Chroma. Pre-chunking processes documents asynchronously by breaking them into smaller pieces before embedding and storing them in the vector database, enabling fast retrieval at query time since all chunks are pre-computed. The index is ready. Nothing happens again until a user asks something.
Step two: turning your question into a search
When your teammate types "What's the on-call escalation path for the payments service?", the system doesn't keyword-search the docs. It embeds the question using the same embedding model, producing a vector for that query.
Chunking is the process of splitting a large piece of text into smaller segments before embedding and storing them in a vector database. When a user asks a question, the system embeds the query and searches for the most semantically similar chunks - not the full documents.
The vector database then runs a nearest-neighbor search - finding the stored chunk vectors closest to the query vector. Cosine similarity is the usual metric: two vectors pointing in nearly the same direction score near 1.0; perpendicular vectors score near 0.
The non-obvious consequence: the model never reads your documents at query time. It reads the top-k chunks the vector search surfaced. If the right chunk wasn't indexed - because it was buried in a PDF that never got ingested, or chunked across a boundary that split the key sentence - the model answers without it. Retrieval failure looks exactly like model failure from the outside.
Production stacks often add a reranking step here. Combining vector search with sparse retrieval (BM25) and applying rerankers for better relevance is a common pattern; in many production workloads, hybrid retrieval improves recall by roughly 1-9% versus pure vector search.
Step three: pasting context into the prompt and generating
The top-k retrieved chunks - say, the three most relevant passages - get inserted into the model's prompt alongside the original question. The final prompt looks roughly like: