What Really Happens When an AI Looks Up Your Docs

Your internal wiki has 5,000 pages. An engineer asks the AI assistant a question. The answer is in there somewhere. What happens next is not a keyword search-it's a four-step pipeline that can fail at any one of those steps, silently, and still hand the model something that looks like a useful answer.

That pipeline is called retrieval-augmented generation, or RAG. It's the mechanism behind almost every "AI that knows your docs" product you'll demo this year. Understanding it at a mechanical level-not a marketing level-changes how you build, configure, and debug one.

What ingestion actually does to your documents

Ingestion is the offline step that makes retrieval possible. It converts raw text into vectors, which are the coordinates the search engine works with.

First, the external knowledge source needs to be prepared-raw data is cleaned and transformed into embeddings through a process called vectorization. But before vectorization, documents are sliced into chunks.

Why slice at all? A chunk is a small unit of content stored and retrieved as a single result. You embed chunks, not whole documents, so retrieval can be precise and the retrieved text can fit into downstream systems.

Chunk size is where most production RAG systems quietly go wrong. Optimal chunk size for RAG systems typically ranges from 128-512 tokens, with smaller chunks excelling at precise fact-based queries while larger chunks provide better context for complex reasoning tasks. In practice, short, focused documents under 500 tokens can hurt retrieval performance when chunked-FAQs, product descriptions, and support tickets often work better embedded as whole documents.

Fixed chunks with no overlap cause semantic fragmentation-sentences split mid-context lose meaning during retrieval. Using 20-50 token overlap keeps related content together.

Once you have chunks, each one gets run through an embedding model. Embedding is the act of running a chunk through an embedding model to produce a dense vector-every chunk becomes a point in a high-dimensional space, where each dimension is a learned feature.

One concrete way to see this: OpenAI's text-embedding-3-small is a 1,536-dimension model optimized for throughput , costing $0.02 per million input tokens . Here's what that means in practice, run through the math: a 5,000-page wiki averaging 1,000 tokens per page is 5 million tokens. Embedding the whole corpus costs roughly $0.10 at ingestion-less than a coffee. The cost to maintain that index as docs update is similarly small; the expensive part is querying, not storing.

A common mistake is changing embedding models without re-embedding the corpus-old and new vectors are incomparable and results silently degrade. Pick your embedding model and stick with it, or version the index explicitly.

How vector similarity search finds the right chunks

Vector search is retrieval based on semantic similarity-"similar" means "close in embedding space." This means it can match "terminate the contract" with "cancel the agreement" even when the words barely overlap.

At query time, the process mirrors ingestion. You embed the query with the same model used for the documents, which means the query and the content live in the same vector space and distances are comparable.

Vector similarity search then computes how close two vectors are using a distance metric such as cosine similarity or dot product. The system returns the top-k nearest neighbors-the chunks most similar to the query.

The non-obvious part: this first pass is fast because it's approximate. Vector search uses a bi-encoder-it turns the query into one vector and each document into one vector separately, then compares those two vectors with cosine similarity. Because the documents are embedded ahead of time, this is blazing fast and works over millions of items.

That speed comes with a cost. In production-grade RAG applications, bi-encoders typically achieve only 65-80% relevance accuracy on complex queries, meaning 20-35% of the information fed to the LLM is noisy or irrelevant.

Why a reranker is a second, completely different model

Reranking is the part most teams skip and then wonder why their AI gives oddly confident wrong answers.

Reranking is the second retrieval stage in a RAG pipeline: a cross-encoder re-scores the top candidates found by the fast bi-encoder and sorts them by genuine query relevance.

A cross-encoder feeds the query and one document together into a transformer, lets every query token attend to every document token, and outputs a single relevance score for that pair. Reading them jointly is what makes it dramatically more accurate-it catches the passage that is superficially similar but actually irrelevant, the bi-encoder's classic mistake.

In practice, hybrid retrieval delivers the top 50-100 candidates, and the reranker reduces them to the most relevant 5-10 for the LLM prompt. You're not running the cross-encoder across your whole index-only across the shortlist the bi-encoder already narrowed down.

What the LLM actually receives, and why generation is the easy part

After retrieval and reranking, the winning chunks get assembled into a prompt. The augmentation process integrates external information by inserting it into a predefined prompt template. After the augmented prompt is injected into the model's context window, the model proceeds to generate the final response.

The LLM combines the retrieved words and its own response to the query into a final answer, potentially citing sources the embedding model found.

This is actually the easiest part of the pipeline to debug: the model is doing in-context learning on whatever text you put in front of it. If the answer is wrong, the problem is almost always upstream-in the chunking strategy, the embedding model choice, or the missing reranking step.

RAG helps reduce AI hallucinations, which have caused chatbots to describe policies that don't exist, or recommend nonexistent legal cases to lawyers looking for citations to support their arguments. But "helps reduce" is doing real work in that sentence. A misconfigured pipeline reduces them less than you'd hope.

RAG in Slack and Teams: common questions

How does retrieval-augmented generation work?

RAG works in four steps: chunk documents into passages, embed each passage into a vector, use semantic similarity at query time to retrieve the closest passages, then hand those passages to the LLM as context. The model answers using that retrieved text rather than what it memorized during training.

Why does my RAG system return irrelevant chunks?

The most common causes are chunks that are too large (mixing topics into one vector), using a different embedding model than the one used to build the index, or skipping the reranking step. A chunk that mixes topics produces a blurred vector that retrieves poorly. Start by auditing what chunks actually reach the prompt.

What is the difference between a bi-encoder and a cross-encoder in RAG?

A bi-encoder turns the query into one vector and each document into one vector, separately, then compares them with cosine similarity.

A cross-encoder feeds the query and one document together into a transformer and outputs a single relevance score for that pair. Bi-encoders are fast enough to search millions of chunks; cross-encoders are accurate enough to pick the best ten.

How much does it cost to embed a large internal knowledge base?

As of mid-2026, text-embedding-3-small costs $0.02 per million input tokens. A 5,000-page wiki at roughly 1,000 tokens per page runs to about $0.10 to embed in full. Re-embedding only happens when docs change or you switch models.

Can RAG replace fine-tuning?

For most knowledge-access use cases, yes. RAG reduces the need to retrain LLMs with new data, saving on computational and financial costs. Fine-tuning changes how a model behaves; RAG changes what it knows. If the problem is stale or private knowledge, RAG is almost always the right tool.