How Vector Embeddings Work in a RAG Pipeline

OpenAI's text-embedding-3-large converts a sentence into a list of 3,072 floating-point numbers. That list is 12 KB per chunk. Store a million of them and you're looking at roughly 12 GB before indexes, replication, or metadata. Most teams plugging embeddings into a RAG pipeline for the first time have no idea they're signing up for that bill-or why 3,072 numbers are needed in the first place.

This post walks through exactly what happens when a RAG pipeline runs: how text becomes a vector, how the vector database finds the right chunks, and what the numbers actually mean. No machine learning background required.

What a vector embedding actually is

An embedding is a fixed-length array of numbers that encodes the meaning of a piece of text. Two sentences that mean roughly the same thing will produce arrays that point in a similar direction in high-dimensional space-even if they share zero words.

text-embedding-3-small and text-embedding-3-large measure the relatedness of text strings. Embeddings are used for search, clustering, recommendations, anomaly detection, and classification-an embedding is a vector (list) of floating point numbers.

The geometry is the key insight. "The server is down" and "our infrastructure is offline" use completely different words, but an embedding model places them close together in vector space because they were trained on text where those phrases appear in similar contexts. A keyword search would miss that match. A vector search catches it.

By default, the embedding vector length is 1,536 for text-embedding-3-small and 3,072 for text-embedding-3-large. Each dimension is one float32 value-4 bytes. A 1,536-dimensional vector uses about 6 KB of raw vector storage, while a 3,072-dimensional vector uses about 12 KB. That arithmetic scales fast: a typical enterprise knowledge base with 10 million documents using text-embedding-3-large (3,072 dimensions at 4 bytes per float32) requires approximately 116 GB of storage just for the embeddings.

How documents get turned into vectors: chunking

Before you can embed a document, you have to split it. Chunking is the act of splitting larger documents into smaller units. Each chunk can be individually indexed, embedded, and retrieved. Because RAG pipelines rely on retrieval from vector databases and LLMs with limited context windows, smart chunking can make all the difference in delivering relevant answers.

The wrong chunk size is one of the most common RAG failures. Factoid queries (specific facts, names, dates) work best with 256-512 tokens-the chunk should contain the answer and minimal surrounding noise. Analytical queries (explanations, comparisons, reasoning) benefit from 1,024+ tokens or page-level chunking, since the LLM needs broader context to synthesize a coherent response.

Overlap handles the boundary problem: if a key sentence lands across two chunk edges, neither chunk retrieves it cleanly. Most RAG systems perform well with chunks between 200 and 500 tokens, with an overlap of 10-20% to preserve context between chunks. The cost of heavy overlap is real- overlapping chunking improves recall by preserving cross-boundary context, but storage increases by a factor of K/(K-O), and redundancy may reduce precision.

A practical rule: recursive character splitting at 400-512 tokens with 10-20% overlap works well for most text content and is the recommended starting point. Page-level chunking performs best for PDFs and paginated documents. Semantic chunking gives higher recall but costs more to run.

How vector search finds the right chunks

Once chunks are embedded and stored, a query arrives. The query text gets embedded by the same model-this is non-negotiable; you must use consistent models and never mix embeddings from different models in the same index. The resulting vector then gets compared against every stored vector to find the closest ones.

Comparing a query vector against millions of stored vectors using exact cosine similarity would be too slow for production. Instead, most vector databases use Approximate Nearest Neighbor (ANN) search-specifically a graph-based algorithm called HNSW (Hierarchical Navigable Small World).

HNSW graphs are among the most widely adopted algorithms for ANN search in high-dimensional data, supporting applications across machine learning, recommendation, and computer vision. The mechanics: it builds a multi-layer graph where the top layers have sparse long-range connections and the bottom layer is dense. A query descends through sparse upper layers quickly-skipping most of the space-then does a careful local search at the bottom. HNSW routinely reaches ~95-99% recall on million-scale datasets with single-digit millisecond latency on CPUs, given sensible parameter choices.

The similarity metric matters. Cosine similarity measures the cosine of the angle between two vectors. It ranges from -1 to 1: a value of 1 means the vectors are identical, 0 means they are orthogonal (no correlation), and -1 means they are completely dissimilar. Cosine ignores vector magnitude-it only cares about direction. When embeddings are normalized to unit length, the dot product and cosine similarity are equivalent. Since most embedding APIs return normalized vectors, dot product is often used in practice because it's cheaper to compute.

As a rule of thumb, use the distance metric that matches the model you're using. Check your embedding model's docs-it will specify which metric it was trained against.

The non-obvious cost of more dimensions

Higher dimensions do not automatically mean better retrieval. Reducing from 3,072 to 1,536 dimensions gives a 50% RAM reduction with only a 1.6% quality loss. Down to 1,024 dimensions delivers a 66% RAM reduction with a 2.8% quality loss-the most practical compromise for most scenarios.

This works because of Matryoshka Representation Learning (MRL). MRL packs the most information toward the front of the vector, so you can remove numbers from the end without the embedding losing its concept-representing properties. A large vector shortened to 256 still beats an unshortened ada-002 at 1,536.

The bigger surprise is what's happened on the MTEB leaderboard. As of April 2026, Qwen3-Embedding-0.6B with 1,024 dimensions scores 70.70 on MTEB, surpassing text-embedding-3-large with 3,072 dimensions at 66.43-with lower dimensionality and full self-hosted availability. Dimensionality ≠ quality.

Vector storage is priced on bytes, determined by: vector count × dimensions × precision. One million 768-dimension float32 embeddings is about 3 GB before indexes. Dimensions matter linearly-3,072-dim embeddings cost 4x what 768-dim ones do to store and search.

How it fits together end-to-end

A complete RAG retrieval call looks like this:

1. Ingest: documents are split into 400-512 token chunks with ~15% overlap
2. Embed: each chunk goes through an embedding API call, returning a vector (e.g., 1,024 floats)
3. Index: vectors are stored in a vector database (Pinecone, Weaviate, pgvector, Qdrant, Milvus) with an HNSW index built on top
4. Query: the user's question is embedded by the same model
5. Search: HNSW finds the top-k closest chunks in single-digit milliseconds with ~97% recall
6. Generate: the retrieved chunks are injected into the LLM's context window as grounding material
7. Respond: the LLM generates an answer citing the retrieved chunks

The retrieval step-steps 4 and 5-is where most production failures happen. Not in the generation. Bad chunking means the right chunk never makes it to the LLM. The wrong embedding model means the query vector lands in the wrong neighborhood entirely. And the wrong dimensions mean you either waste money on storage or lose recall you didn't know you had.

A teammate like Beagle sitting in Slack runs this pipeline in the background every time someone asks a question that could be answered from internal docs-Notion, Confluence, Google Docs-and surfaces the closest chunk with a source link before posting.

How vector embeddings work: common questions

What is a vector embedding in simple terms?

A vector embedding is a list of numbers that represents the meaning of a piece of text. Two pieces of text with similar meanings produce similar number lists. The model learns this mapping during training, so "server is down" and "infrastructure is offline" end up close together even with no shared words.

How many dimensions does an embedding need?

It depends on the model and use case. OpenAI's text-embedding-3-small defaults to 1,536 dimensions; text-embedding-3-large to 3,072. For most RAG workloads, 1,024 dimensions delivers ~97% of full-size quality at one-third the storage cost, using the Matryoshka truncation parameter.

Why does chunk size matter for RAG retrieval?

Chunk size determines what the embedding model sees. A chunk that's too small may not contain enough context to embed meaningfully. A chunk that's too large dilutes the signal. Most RAG systems perform well at 400-512 tokens with 10-20% overlap as a starting point, tuned to the specific query types expected.

What is HNSW and why do vector databases use it?

HNSW (Hierarchical Navigable Small World) is the most widely used algorithm for fast approximate nearest-neighbor search. It builds a layered graph over all vectors, letting a query skip most of the search space and land near the right answer in milliseconds. It reaches 95-99% recall on million-scale datasets without checking every stored vector.

Does a higher-dimensional embedding model give better retrieval?

Not necessarily. As of April 2026, Qwen3-Embedding-0.6B at 1,024 dimensions outscores OpenAI's text-embedding-3-large at 3,072 dimensions on MTEB. More dimensions cost linearly more in storage and search latency. Pick the smallest dimension that meets your recall target-benchmark on your own data, not just published leaderboards.