Cosine Similarity in RAG: How Vector Retrieval Actually Works

Cosine similarity is the default scoring function inside almost every retrieval-augmented generation (RAG) pipeline shipped in the last three years. When a user asks a question, the system embeds the query, compares that vector against a store of document embeddings, and returns the top-k closest matches to feed into the language model. That comparison step is nearly always cosine similarity - and yet most teams building RAG systems treat it as a black box.

This article explains what cosine similarity actually is, why it became the standard for semantic search, how it behaves in production RAG pipelines, and when you should reach for something else. It assumes you already know roughly what RAG is; if you need a refresher, the retrieval step is where cosine similarity earns its place.

What cosine similarity measures

Cosine similarity measures the angle between two vectors in high-dimensional space. It ignores their magnitude entirely and looks only at direction. The formula is straightforward:

cos(θ) = (A · B) / (||A|| × ||B||)

Where A · B is the dot product of the two vectors and ||A|| is the Euclidean norm (length) of vector A. The result is bounded between -1 and 1. A score of 1 means the two vectors point in exactly the same direction, 0 means they are orthogonal (unrelated), and -1 means they point in opposite directions. In practice, with modern text embeddings, scores below 0 are rare because embedding models tend to produce vectors that cluster in a narrow region of the unit sphere.

The key intuition: cosine similarity treats the pattern of features in a vector as meaningful, and the overall size as noise. Two documents can have very different lengths, word counts, and embedding magnitudes but still be semantically identical - cosine similarity captures that. A short summary and a long article about the same topic will score highly, whereas Euclidean distance would penalise the length difference.

Why embeddings and cosine similarity go together

Modern embedding models - OpenAI's text-embedding-3-large, Cohere's embed-v3, Voyage AI's voyage-3, open-source models like BGE and E5 - convert text into dense vectors of 384 to 3,072 dimensions. Each dimension is a learned feature, and the model is trained so that texts with similar meaning produce vectors pointing in similar directions.

Crucially, most embedding models are trained with a contrastive loss objective that explicitly optimises for cosine similarity. Sentence-BERT, the paper that kicked off the modern embedding era in 2019, used cosine similarity as its target metric during training. OpenAI's embedding models are explicitly documented as optimised for cosine similarity - their embeddings guide notes that all their embedding models are normalised to length 1, which makes cosine similarity and dot product equivalent.

This matters. If your embedding model was trained with cosine similarity as its objective, using cosine similarity at retrieval time is not a design choice - it is the model's native tongue. Switching to Euclidean distance or another metric would be like asking a bilingual speaker questions in a language they only half-know.

Cosine similarity vs dot product vs Euclidean distance

These three metrics come up constantly in RAG documentation, and vector databases like Pinecone, Weaviate, Qdrant, and pgvector let you choose between them. The relationship is worth understanding.

Dot product is the numerator of the cosine similarity formula. It measures both direction and magnitude. If your vectors are normalised to length 1 (unit vectors), dot product and cosine similarity produce identical rankings - the denominator becomes 1×1=1 and drops out. This is why OpenAI's docs recommend dot product for their embeddings: it is faster to compute and mathematically equivalent when vectors are pre-normalised.

Euclidean distance (L2) measures the straight-line distance between two points in space. It cares about both direction and magnitude. For non-normalised vectors, L2 will treat two similar documents of different lengths as further apart than they really are. For normalised vectors, L2 and cosine similarity produce equivalent rankings but with different absolute values - the relationship is L2² = 2 - 2×cos(θ).

Practical guidance: if your embedding model produces normalised vectors (most modern ones do), use dot product for speed. If you are unsure whether your vectors are normalised, use cosine similarity - it normalises on the fly at the cost of two extra square-root operations per comparison. Only use Euclidean distance if you have a specific reason, such as working with embeddings from a model that was explicitly trained with L2 loss, which is unusual for text.

How cosine similarity behaves in a real RAG pipeline

In production, cosine similarity is rarely the only thing deciding what context reaches the LLM. A typical retrieval stage looks like this:

1. Query arrives, gets embedded to a vector.
2. Vector database performs approximate nearest neighbour (ANN) search - usually HNSW or IVF - to return the top 50-200 candidates ranked by cosine similarity.
3. A reranker (Cohere Rerank, Voyage Rerank, or a cross-encoder like BGE-reranker) scores those candidates using a slower, more accurate model.
4. The top 5-15 reranked chunks go to the LLM.

Cosine similarity does the heavy lifting in step 2, where speed matters more than precision. Its job is to be roughly right very fast - to narrow millions of documents down to a manageable candidate set. The reranker then applies more expensive reasoning about actual query-document relevance.

The interesting behaviour: cosine similarity scores in production RAG systems tend to cluster tightly. It is common to see the top 100 results all scoring between 0.72 and 0.85, with no clean threshold separating relevant from irrelevant. This is the anisotropy problem: embedding models push most vectors into a narrow cone of the vector space, so almost everything looks somewhat similar to everything else. Research from Ethayarajh (2019) and others has documented this consistently.

The practical consequence: do not use absolute cosine scores as a relevance threshold without calibration. A score of 0.78 might be excellent for one query and mediocre for another. Rank-based thresholds (top-k) and reranker-based filtering work better than fixed similarity cutoffs.

Common pitfalls when using cosine similarity for RAG

Several failure modes come up repeatedly in RAG projects, and most of them are misdiagnosed as problems with the language model when the retrieval stage is at fault.

Chunk-size mismatch. If you embed 2,000-word chunks and the user asks a specific factual question, the embedding of the chunk gets averaged across dozens of topics and the cosine score to a targeted query drops. Smaller chunks (200-500 tokens) usually improve cosine relevance for factual Q&A. Larger chunks help for summarisation.

Query-document asymmetry. A short query like "refund policy" produces a different embedding shape than a long document paragraph. Some embedding models (BGE, E5, Voyage) offer asymmetric encoding with separate query and document prefixes to correct for this. Using symmetric encoding with asymmetric content silently degrades cosine similarity rankings.

Multilingual and code content. General-purpose embedding models handle English prose well, code passably, and non-English content variably. Cosine similarity on out-of-distribution content is close to random. Test empirically on your actual content mix before assuming the metric works.

Vocabulary vs semantic overlap. Cosine similarity on dense embeddings captures semantic similarity but can miss exact-match requirements. A query for a specific SKU code, part number, or legal citation may score higher against a paragraph discussing similar concepts than against the paragraph that actually contains the identifier. Hybrid search - combining cosine similarity on embeddings with BM25 keyword scoring - fixes this. Weaviate, Elastic, OpenSearch, and Qdrant all support hybrid retrieval natively.

Stale embeddings. If you re-index with a new embedding model, all your cosine scores become incomparable to historical scores. This trips up teams who cache retrieval results or build monitoring dashboards on absolute similarity thresholds.

When to use something other than cosine similarity

Cosine similarity is the correct default for text RAG. There are legitimate reasons to reach for alternatives, though.

Use dot product without normalisation when your embedding model deliberately encodes importance or confidence in vector magnitude. Some domain-specific embedding models do this. Rare in general text RAG.

Use BM25 or hybrid search when your queries are keyword-heavy, contain named entities, product codes, or legal citations, or when your users behave more like search engine users than natural-language questioners. Microsoft's research on Bing and various enterprise search deployments consistently show that hybrid retrieval beats pure vector retrieval on real user queries.

Use a cross-encoder reranker when precision at the top of the list matters more than latency. Cross-encoders score query-document pairs jointly and are significantly more accurate than any bi-encoder cosine similarity, at the cost of running one model call per candidate document. Cohere Rerank v3 and BGE-reranker-v2 are strong choices.

Use a knowledge graph or structured retrieval when your data has hard relational structure that embeddings flatten out. Compliance documents with cross-references, product catalogues with taxonomies, and codebases with dependency graphs all benefit from structured retrieval alongside or instead of vector search.

Tuning cosine similarity retrieval in practice

If cosine similarity is underperforming in your RAG system, the levers to pull, roughly in order of impact:

1. Change the embedding model. Moving from OpenAI text-embedding-ada-002 (2022) to text-embedding-3-large or Voyage voyage-3 typically improves retrieval quality by 10-25% on standard benchmarks like MTEB. The metric is the same; the vectors are better.
2. Fix chunking. Try 300-token chunks with 50-token overlap as a starting point. Measure recall@10 on a labelled evaluation set before and after.
3. Add a reranker. This is the single highest-impact change for most underperforming RAG systems. Expect 15-40% improvement in top-3 relevance.
4. Add hybrid search. Combine cosine similarity on embeddings with BM25 keyword scores, typically using reciprocal rank fusion or a weighted sum.
5. Consider fine-tuning the embedding model on your domain if you have thousands of labelled query-document pairs. This shifts the vector space so cosine similarity captures your specific notion of relevance more precisely.

Whatever you change, you need an evaluation harness. Cosine similarity is a mathematical operation; retrieval quality is an empirical question. Build a set of 50-200 representative queries with known-correct answers, measure recall and precision, and re-run the harness on every change. Without this, tuning becomes vibes-based and regressions creep in unnoticed.

Frequently asked questions

Is cosine similarity always the best metric for RAG?

Cosine similarity is the best default for text RAG when your embedding model was trained with a cosine or contrastive objective, which covers nearly all modern text embedding models. It is not universally best. For queries dominated by keywords, named entities, or product identifiers, hybrid search combining cosine similarity with BM25 keyword scoring outperforms pure cosine retrieval. For very high-precision requirements at the top of the results list, a cross-encoder reranker applied on top of cosine similarity candidates outperforms cosine alone by a wide margin. Treat cosine similarity as a strong default, not a final answer.

What is a good cosine similarity score for RAG retrieval?

There is no universal threshold. Modern text embedding models suffer from anisotropy, meaning most vectors cluster in a narrow region of the vector space and produce scores in a compressed range - often 0.6 to 0.9 for anything vaguely on-topic. A score of 0.85 might be excellent for one query and mediocre for another. Do not hardcode a relevance threshold like "only return results above 0.75" without calibration on your specific content and queries. Use rank-based selection (top-k) and rely on a reranker or evaluation harness to determine actual relevance.

How does cosine similarity handle long documents versus short passages?

Cosine similarity itself is length-invariant, which is one of its advantages over Euclidean distance. However, the embedding step usually is not. Long documents embedded as a single vector average many topics together, diluting the signal for any specific query. Short passages produce more focused embeddings. This is why practically all RAG systems chunk documents into passages of a few hundred tokens rather than embedding whole documents. The cosine similarity between a query and a well-scoped chunk will typically be much higher than between the same query and a full document containing that chunk.

Do I need to normalise my vectors before using cosine similarity?

The cosine similarity formula normalises vectors internally as part of the calculation, so you get the same result whether you pre-normalise or not. However, if you pre-normalise your vectors to unit length, dot product becomes mathematically equivalent to cosine similarity and runs faster because it skips the division. Most modern embedding APIs return normalised vectors by default - OpenAI's embeddings are always unit length, for example. Check your model's documentation. If vectors are already normalised, configure your vector database to use dot product for a small performance win.

Why do my RAG results look wrong even though cosine similarity scores are high?

High cosine similarity means the query and retrieved chunk have similar semantic embeddings. It does not mean the chunk answers the query. A question like "how do I cancel my subscription" will score highly against a paragraph titled "how to sign up for a subscription" because both are about subscriptions. This is the classic reason to add a reranker: cross-encoders evaluate actual query-answer relevance rather than embedding similarity. If your retrieval scores look right but the LLM's answers are wrong, the retrieval stage is probably surfacing topically related but not actually relevant content.

Can I use cosine similarity for images, audio, or multimodal RAG?

Yes, and it works well. CLIP, Google's SigLIP, ImageBind, and other multimodal embedding models all produce vectors intended to be compared with cosine similarity. The same principles apply: verify your embedding model was trained with cosine as its objective, normalise vectors if you want the dot product performance boost, and use a reranker or evaluation harness for anything requiring precision. Multimodal retrieval tends to benefit even more from hybrid approaches because visual and audio embeddings can be noisy for specific attribute queries.

How much does cosine similarity computation cost at scale?

Naive cosine similarity is O(d) per comparison, where d is the vector dimension - trivial for a single pair. The cost problem is at scale: comparing a query against 10 million vectors of 1,536 dimensions each is 15 billion floating-point operations. Vector databases avoid this by using approximate nearest neighbour indexes like HNSW or IVF, which reduce query time to O(log n) at the cost of occasionally missing the true top-k. For most RAG workloads on tens of millions of chunks, expect single-digit millisecond retrieval latency using Pinecone, Qdrant, Weaviate, or pgvector with an HNSW index.

Getting retrieval right in production

Cosine similarity is a simple mathematical operation doing a very well-defined job inside a much more complex system. Understanding it properly lets you stop treating retrieval as a mysterious black box and start diagnosing why your RAG pipeline is or is not working. The wins in production RAG almost never come from replacing cosine similarity with a cleverer metric; they come from better embeddings, better chunking, hybrid search, and a reranker on top.

If you are building a RAG system and finding that retrieval quality is holding back the whole application, AI Advisory helps mid-market teams design and ship production-grade RAG pipelines with proper evaluation harnesses, hybrid retrieval, and reranking built in from day one.

Ready to put this into production? book a discovery call.