Cosine Similarity in RAG: What It Is and When to Use It

Every retrieval-augmented generation system rests on the same quiet assumption: that we can measure how similar two pieces of text are by comparing vectors. In most RAG stacks, the specific measure doing that work is cosine similarity. It is the default in Pinecone, Weaviate, Qdrant, Chroma, pgvector, and almost every tutorial you will read. It is also frequently misunderstood, occasionally the wrong choice, and increasingly not the only thing you should be doing.

This article explains what cosine similarity actually is, why it became the default for semantic retrieval, when it breaks down, and what a production-grade RAG retrieval strategy looks like once you get past the tutorial stage.

What cosine similarity actually measures

Cosine similarity is the cosine of the angle between two vectors. If two vectors point in exactly the same direction, cosine similarity is 1. If they are perpendicular, it is 0. If they point in opposite directions, it is -1. In practice, embeddings from models like OpenAI's text-embedding-3-large or Cohere's embed-v3 produce vectors that mostly sit in a positive region, so real-world cosine scores usually land between about 0.3 and 0.95.

The formula is straightforward: take the dot product of the two vectors, then divide by the product of their magnitudes. The division is the important bit. It normalises for length, which means a long document and a short query can still be judged similar based purely on their direction in the embedding space. That property is why cosine similarity dominates text retrieval - documents vary wildly in length, and you do not want length to be the thing that decides relevance.

Under the hood, when the vectors are already normalised to unit length (which most embedding providers do for you, and which every serious vector database does at index time), cosine similarity becomes mathematically identical to the dot product. This is why you will see vector databases offering both metrics interchangeably: on normalised vectors, they return the same ranking. Dot product is cheaper to compute, which is why Meta's FAISS library and most production systems use it internally.

Why RAG uses cosine similarity by default

RAG systems retrieve chunks of text to feed into a large language model as context. The retrieval step needs to answer: given this user query, which chunks of my knowledge base are most likely to contain the answer? Cosine similarity became the default answer for four practical reasons.

First, embedding models are trained to make it work. Models from OpenAI, Cohere, Voyage, and the open-source sentence-transformers family are trained with contrastive objectives that explicitly optimise for cosine similarity between semantically related pairs. The original Sentence-BERT paper by Reimers and Gurevych, which underpins much of modern semantic search, was designed around this metric. Using anything else usually degrades retrieval quality because you are measuring on a scale the model was not trained to produce.

Second, it is length-invariant. A three-sentence answer and a ten-page document can both be judged relevant to the same query without one being penalised for verbosity or the other rewarded for brevity. In a corpus where chunk sizes vary - which is almost every real corpus - this matters.

Third, it is computationally cheap. Modern vector indexes use approximate nearest neighbour algorithms like HNSW (Hierarchical Navigable Small World graphs) or IVF (Inverted File Index) that can search millions of vectors in milliseconds when the metric is cosine or dot product. Other metrics, particularly ones that require full recomputation, do not benefit from the same index structures.

Fourth, it is bounded and interpretable. Scores sit between -1 and 1, which makes thresholding tractable. You can say "retrieve everything above 0.75" and reason about what that means. Euclidean distance, by contrast, is unbounded and depends on the embedding dimension, which makes threshold-setting harder.

Cosine similarity versus the alternatives

Three metrics dominate vector search: cosine similarity, dot product, and Euclidean (L2) distance. Understanding when each applies stops you making bad defaults.

Dot product is what you actually want if you care about both direction and magnitude. In some retrieval scenarios, longer or more information-dense passages should score higher, and the magnitude of the embedding carries that signal. Models like OpenAI's older text-embedding-ada-002 were designed to be used with dot product, and normalising away magnitude would throw away useful information. In practice, if your vectors are normalised, dot product and cosine give identical rankings, so the choice is about whether to normalise at index time.

Euclidean distance measures the straight-line distance between two points in the embedding space. It is sensitive to magnitude, which is usually a bug rather than a feature for text embeddings. It is the right choice for image embeddings from some architectures, for clustering with algorithms like k-means, and for embeddings where absolute position in the space carries semantic meaning. For text RAG, it is almost never the right default.

Manhattan (L1) distance and other exotic metrics show up occasionally in specialised systems but rarely in general-purpose RAG. If you find yourself reaching for them, the more likely fix is a better embedding model, not a different distance function.

The pragmatic guidance: use cosine similarity (or equivalently, dot product on normalised vectors) unless your embedding provider explicitly recommends otherwise. Both OpenAI's embeddings documentation and Cohere's guidance point to cosine similarity as the default for their current text embedding models.

Where cosine similarity breaks down in RAG

The uncomfortable truth is that cosine similarity is a coarse instrument. It measures semantic overlap in the abstract, but semantic overlap is not the same as "contains the answer to this query." Several failure modes show up predictably in production RAG systems.

Keyword-heavy queries. If a user searches for a specific product SKU, error code, or proper noun, dense embeddings often fail to prioritise exact matches. The embedding for "error E-4471" and "error E-4472" are cosine-similar, but they refer to entirely different problems. This is why hybrid search - combining dense vector retrieval with sparse lexical methods like BM25 - has become standard practice. Elastic, Weaviate, Qdrant, and pgvector all now support hybrid retrieval natively.

Negation and logical structure. Cosine similarity treats "the policy covers water damage" and "the policy does not cover water damage" as extremely similar because they share most of their tokens and topic. For any RAG system operating on legal, medical, financial, or insurance content, this is dangerous. The mitigation is not a different distance metric but a re-ranking step, typically using a cross-encoder model.

Multi-hop questions. Questions that require synthesising information from multiple documents rarely get answered by top-k cosine retrieval alone. The relevant chunks may each score moderately rather than exceptionally, and the truly relevant chunk for step two of the reasoning may not surface until step one has been answered. Techniques like HyDE (Hypothetical Document Embeddings), query decomposition, and iterative retrieval address this - the underlying similarity metric does not.

Domain shift. Off-the-shelf embedding models are trained on general web text. If your corpus is medical trial protocols, or SAP transaction codes, or Scottish property law, the general model may not distinguish between concepts that are semantically distinct in your domain. Fine-tuning an embedding model on domain pairs, or using a specialised model like BioBERT or LegalBERT, often produces larger retrieval improvements than any change to the similarity metric.

Chunk boundaries. Cosine similarity is only as good as the chunks it compares. If your chunking strategy splits a definition from its explanation, or an instruction from its precondition, no distance metric will save you. Chunking - by semantic boundaries, by structural markers, with overlap - deserves more engineering attention than metric choice in almost every RAG project we ship.

What a production RAG retrieval pipeline actually looks like

A retrieval pipeline that relies on cosine similarity alone is a prototype. The pattern for production systems that survive contact with real users has settled into a fairly standard shape.

Stage one: hybrid retrieval. Run the query through both a dense vector search (cosine similarity against embeddings) and a sparse lexical search (BM25 or SPLADE). Combine the results using reciprocal rank fusion or a weighted score. This typically retrieves 40-100 candidates. Microsoft's research on their Azure AI Search stack, and public write-ups from Anthropic on contextual retrieval, both make this case.

Stage two: re-ranking. Feed those candidates plus the query into a cross-encoder model - Cohere's Rerank, Voyage's rerankers, or an open model like BGE-reranker. Cross-encoders are far more accurate than bi-encoder cosine similarity because they attend to the query and document jointly, but they are too expensive to run at index scale. Running them on 40-100 candidates is affordable and often lifts retrieval precision by 10-30 percentage points on realistic benchmarks.

Stage three: contextual filtering. Apply business rules - recency, source authority, access permissions, jurisdiction. This is where GDPR data-subject boundaries, tenant isolation, and content governance live. Under the UK GDPR, if your RAG system surfaces personal data, this filtering step is where you enforce lawful-basis constraints.

Stage four: context assembly. Select the final top-k chunks (usually 3-10 depending on model context window and cost budget), deduplicate near-identical passages, and format them for the generation model. Anthropic's contextual retrieval work suggests prepending short chunk-level context summaries can further reduce hallucination.

Cosine similarity does one job in this pipeline: the first-pass dense retrieval in stage one. It is important, but it is roughly 20% of what makes retrieval work.

Practical implementation notes

A few implementation details make the difference between a RAG system that works in a demo and one that works in production.

Normalise vectors at index time. Every mature vector database does this by default when you select cosine similarity as the metric. If you are using pgvector directly, either normalise before insertion or use the <=> cosine distance operator, which handles it for you. Do not mix normalised and un-normalised vectors in the same index.

Do not confuse similarity and distance. Cosine similarity ranges from -1 to 1, higher is better. Cosine distance is 1 minus similarity, ranging from 0 to 2, lower is better. Databases variously return one or the other, and mixing them up produces reversed rankings that are surprisingly hard to debug.

Evaluate with real queries. Cosine similarity scores are not a proxy for retrieval quality. Build an evaluation harness with 50-200 real queries and known-good answers, and measure recall@k, MRR (mean reciprocal rank), or nDCG. Frameworks like Ragas and TruLens make this tractable. Ship changes based on evaluation metrics, not on how good the top score looks.

Watch embedding drift. If you change embedding models, you must re-index everything. Mixing embeddings from different models in the same index produces cosine scores that are technically valid but semantically meaningless. Version your embeddings alongside your model choice.

Budget for the whole pipeline. Embedding costs are usually trivial. Re-ranker inference costs and generation costs dominate. When we cost RAG systems for clients, first-pass retrieval is typically 2-5% of the runtime cost. Optimising cosine similarity computation is almost never where the wins are.

Frequently asked questions

Is cosine similarity the same as semantic similarity?

No. Cosine similarity is a mathematical operation on two vectors. Semantic similarity is a property of the underlying texts. Cosine similarity is a good proxy for semantic similarity only because the embedding model that produced the vectors was trained to make it one. If you pass random vectors through cosine similarity, you get meaningless numbers. If you use a poorly trained or domain-mismatched embedding model, the cosine scores will look reasonable but will not reflect actual semantic relatedness in your domain. The metric is only as good as the embeddings.

Should I use cosine similarity or dot product for my RAG system?

For virtually all modern text embedding models, the two produce identical rankings because the vectors are normalised to unit length either by the model or by the vector database. Choose based on what your embedding provider recommends and what your vector database uses natively. OpenAI's current embedding models, Cohere's embed-v3, and most open sentence-transformer models are designed for cosine similarity. If you are using a database that internally uses dot product on normalised vectors (which most do for performance), you are effectively using cosine similarity regardless of which name appears in the API.

What cosine similarity score means a chunk is relevant?

There is no universal threshold. Scores depend on the embedding model, the domain, the chunking strategy, and the query style. A score of 0.85 might be a strong match with one model and mediocre with another. The correct approach is to build an evaluation set of real queries with known-relevant chunks, then empirically determine the threshold that maximises whatever metric you care about (recall, precision, F1). Expect thresholds in the 0.3-0.5 range for OpenAI's text-embedding-3 models on general content, and higher for models that produce more compressed embedding spaces. Never hardcode a threshold from a tutorial.

Why does my RAG system return irrelevant chunks with high cosine similarity?

Usually one of four reasons: the chunks contain topical overlap without answering the specific question (common with negation or qualification); the embedding model does not distinguish concepts that matter in your domain; your chunking strategy has split answers away from their context; or you are relying on dense retrieval alone without lexical fallback or re-ranking. The fix is rarely a different similarity metric. It is hybrid retrieval, a cross-encoder re-ranker, better chunking, or a domain-adapted embedding model. Diagnose by manually inspecting the top-20 candidates for failing queries and asking what would need to change for the right chunk to win.

Do I need a vector database, or can I compute cosine similarity myself?

For up to about 100,000 chunks, computing cosine similarity in-memory with NumPy or FAISS on a single server works fine and returns results in tens of milliseconds. For millions of chunks, or when you need concurrent queries, filtered search, or high availability, a dedicated vector database (Pinecone, Weaviate, Qdrant, Milvus) or a vector-capable general database (pgvector, Elasticsearch, MongoDB Atlas) is worth the operational overhead. The similarity computation is the same; what you are paying for is indexing, filtering, replication, and query planning at scale.

How is cosine similarity different from BM25?

BM25 is a lexical scoring function. It counts term overlaps between query and document, weighted by term frequency and inverse document frequency. It matches on exact words and their variants. Cosine similarity on embeddings measures semantic proximity in a learned vector space and can match "car" with "automobile" or "UK" with "Britain" even without shared tokens. Neither is universally better. BM25 wins on rare terms, proper nouns, and exact-phrase queries. Cosine wins on paraphrase, synonym, and concept-level matching. Modern RAG systems combine both, which is why hybrid search has become the production default.

Does cosine similarity work for multilingual RAG?

Yes, provided you use a multilingual embedding model like Cohere's embed-multilingual-v3, OpenAI's text-embedding-3, or an open model like BGE-M3. These models are trained so that semantically equivalent text in different languages produces vectors with high cosine similarity. A query in English can retrieve a French document about the same topic. The metric itself is language-agnostic; the linguistic capability lives entirely in the embedding model. Do not use monolingual English embeddings for multilingual corpora and expect cross-lingual retrieval to work.

Where this leaves you

Cosine similarity is the sensible default for RAG retrieval, and understanding it well matters. But treating it as the interesting problem is a mistake. The wins in production RAG come from hybrid retrieval, cross-encoder re-ranking, evaluation-driven iteration on chunking, and domain-adapted embeddings. The distance metric is the last thing to tune and the first thing tutorials over-emphasise.

If you are building a RAG system and want a second opinion on your retrieval architecture, evaluation strategy, or chunking approach, AI Advisory's team designs and ships production RAG for UK mid-market businesses every week. Get in touch to talk through what you are building.

Ready to put this into production? book a discovery call.