What Is an Embedding in RAG? A Practical Explainer

Retrieval-augmented generation (RAG) systems live or die on one thing: whether the right chunk of source material reaches the language model at query time. Embeddings are the mechanism that makes that retrieval possible. If you are building a RAG system, or evaluating one built by a vendor, you need a working mental model of what embeddings are, how they are generated, and where they quietly go wrong.

This article explains embeddings in the specific context of RAG: what they represent, how they are produced, how similarity search uses them, which embedding models are worth considering in 2026, and the failure modes that catch teams out in production.

The one-sentence definition

An embedding is a fixed-length list of numbers - a vector - that represents the meaning of a piece of text in a way a computer can compare to other pieces of text. For a RAG pipeline, you generate an embedding for every chunk of your source documents, store those vectors in a database, and at query time generate an embedding for the user's question. The database returns the chunks whose vectors are numerically closest to the question's vector, and those chunks are handed to the LLM as context.

The core assumption is that texts with similar meaning produce similar vectors. "How do I reset my password?" and "I forgot my login credentials, what now?" will land close together in the vector space, even though they share almost no vocabulary. That is the property that makes RAG work at all.

What a vector actually looks like

A modern embedding is typically a list of between 384 and 3,072 floating-point numbers. OpenAI's text-embedding-3-small produces 1,536-dimensional vectors by default. Cohere's embed-v4.0 supports up to 1,536. Open-source models such as BGE-M3 output 1,024 dimensions. The number itself is not important - what matters is that every text you embed with the same model produces a vector of the same length, so they can be compared.

Each dimension does not correspond to a human-readable concept. You cannot point at position 47 and say "this measures how legal the text sounds". The dimensions emerge from training on billions of text pairs, and the meaning is distributed across the whole vector. This is why embeddings feel like a black box: they work, they are reproducible, but the internal structure is not interpretable in the way a decision tree is.

The comparison operation is almost always cosine similarity: the cosine of the angle between two vectors. A cosine of 1.0 means identical direction (very similar meaning), 0.0 means unrelated, and -1.0 means opposite. In practice, useful matches tend to sit above 0.7 for well-tuned models, but the absolute number varies by model and domain - you calibrate thresholds empirically, not by folklore.

How embeddings are generated

An embedding model is a neural network - almost always a transformer - trained specifically to map text to vectors such that semantically similar inputs produce numerically similar outputs. Training uses techniques like contrastive learning: the model sees pairs of texts labelled as similar or dissimilar (query and relevant document, question and correct answer, paraphrases of the same sentence) and learns to pull the similar pairs together in vector space and push dissimilar ones apart.

At inference time, you send text to the model's API (or run it locally) and receive the vector back. A single call to OpenAI's embedding endpoint returns embeddings for a batch of inputs in a few hundred milliseconds. Costs are low: text-embedding-3-small is $0.02 per million tokens as of writing, so embedding a corpus of one million typical documents (say, 500 tokens each) costs around $10.

Two practical constraints:

• Context window. Each embedding model has a maximum input length, usually 8,192 tokens. Longer texts must be chunked before embedding.
• Model consistency. Once you embed a corpus with a given model, every future query must use the same model. Switching models means re-embedding everything.

Chunking: the decision that dominates retrieval quality

Before you embed anything, you have to decide how to split your source documents into chunks. This is the single most under-appreciated decision in RAG engineering. Chunk too small and you lose context - the model retrieves a sentence that mentions the right thing but does not explain it. Chunk too large and you dilute meaning - the embedding averages across multiple topics, and retrieval becomes fuzzy.

Sensible defaults for prose documents sit between 300 and 800 tokens per chunk, with 10-15% overlap between adjacent chunks so a concept that straddles a boundary is captured in both. But defaults are a starting point, not an answer. Legal contracts benefit from clause-level chunking. Technical documentation often works best chunked by heading structure. Chat transcripts should usually chunk by turn or by conversation.

The rule of thumb we apply on client builds: a chunk should be the smallest self-contained unit that answers a plausible question on its own. If a reader would need to look at the paragraph above or below to make sense of the chunk, the chunk is too small or badly aligned to the document's structure.

Choosing an embedding model in 2026

The embedding model market has consolidated around a handful of credible options. The Massive Text Embedding Benchmark (MTEB), maintained on Hugging Face, is the standard reference for comparing them across retrieval, classification, and clustering tasks. It is worth checking directly rather than trusting vendor marketing.

The realistic shortlist for most builds:

• OpenAI text-embedding-3-small and -3-large. Cheap, fast, well-supported, good general performance. The default choice if you have no strong reason to go elsewhere.
• Cohere Embed v4. Strong multilingual performance, competitive on English, native support for reranking as a follow-up step.
• Voyage AI (voyage-3, voyage-code-3). Consistently near the top of MTEB retrieval leaderboards, with domain-specific variants for code and legal text.
• BGE-M3, E5, Nomic Embed. Open-source models you can self-host. BGE-M3 in particular supports dense, sparse, and multi-vector retrieval from the same model, which is genuinely useful.

For UK organisations with data residency requirements under UK GDPR, self-hosted open-source models remove the third-party processing question entirely. The ICO's guidance on AI and data protection is the reference point here: sending customer data to a US-based embedding API is a processing activity that needs its own lawful basis and, usually, a data processing agreement.

Domain matters more than benchmark position. A general-purpose model that scores 65 on MTEB may still outperform a domain-specialist model that scores 68 on your actual corpus of insurance claims. Always evaluate on a sample of your own data with a set of realistic queries before committing.

Vector databases and similarity search

Once you have embeddings, you need somewhere to store them and something to search them fast. Options fall into three groups:

• Postgres with pgvector. Our default for most client builds. If you already run Postgres, adding vector search costs almost nothing operationally, and the performance is more than adequate up to tens of millions of vectors with the HNSW index.
• Dedicated vector databases - Pinecone, Weaviate, Qdrant, Milvus. Better ergonomics for very large corpora (hundreds of millions of vectors) or for teams who want managed infrastructure. Extra moving part, extra cost.
• Search engines with vector support - Elasticsearch, OpenSearch, Typesense. Sensible if you already run one for full-text search and want hybrid retrieval in a single system.

Similarity search at scale uses approximate nearest neighbour (ANN) algorithms, most commonly HNSW (Hierarchical Navigable Small World). Exact search compares the query vector to every stored vector - fine up to a few hundred thousand documents, painful beyond. HNSW trades a small amount of recall accuracy (typically 95-99% of exact results) for orders of magnitude faster queries. Pinecone's technical writeup on HNSW is the clearest explanation in circulation.

Where embeddings quietly fail

Embeddings are not magic and they have known failure modes. If you are running a RAG system in production, these are the ones to watch:

Lexical mismatch on named entities. Embedding models are trained on general text and can be surprisingly weak at distinguishing between two similarly-named products, people, or codes. A query for "policy AB-4471" may retrieve chunks about "policy AB-4741" because the embedding treats the code as a fuzzy string. The fix is hybrid retrieval: combine vector search with keyword search (BM25) and merge results. Almost every serious RAG system in production uses hybrid retrieval, not pure vector search.

Domain drift. A model trained mostly on web text will underperform on medical notes, legal drafting, or engineering specifications. Symptoms: retrieval feels vaguely right but keeps missing the specific document you need. Fix: switch to a domain-tuned model, or fine-tune an open-source model on your corpus. Fine-tuning an embedding model with a few thousand query-document pairs is achievable and often gives a 5-15 point retrieval improvement.

Chunk-query granularity mismatch. Users ask short questions; your chunks are long paragraphs. The embedding of a 600-token chunk is an average of many concepts, while the query embedding is sharp. Retrieval degrades. Fix: generate synthetic questions for each chunk during ingestion and embed the questions, not just the chunk text. Or use a multi-vector approach where each chunk is represented by several embeddings (title, summary, full text).

Stale embeddings. Your source documents change; your vector store does not. This is an operations problem, not a modelling one, but it is where most production RAG systems degrade over time. Set up an ingestion pipeline with change detection from day one, not as a fix later.

Reranking as a second stage. Even a well-tuned retrieval step returns a mix of good and mediocre chunks in its top 20. A cross-encoder reranker (Cohere Rerank, BGE reranker, Voyage rerank) re-scores the shortlist by looking at query and chunk together, and typically lifts answer quality more than any other single change. If you are not reranking, that is usually the first upgrade to make.

Putting it together: a minimal RAG retrieval flow

To make the pieces concrete, here is what a working retrieval pipeline looks like end to end:

1. Ingest documents. Parse to plain text, preserving structure.
2. Chunk. Split into 400-600 token chunks with 15% overlap, aligned to natural boundaries.
3. Enrich. Optionally generate a title, summary, and 3-5 synthetic questions per chunk.
4. Embed. Send each chunk (and each synthetic question) to the embedding model. Store the vectors alongside metadata in Postgres + pgvector or your vector store of choice.
5. Index. Build an HNSW index for approximate nearest neighbour search.
6. At query time: embed the user's question with the same model. Run vector search for the top 20-50 candidates. Run BM25 keyword search in parallel and merge (hybrid retrieval).
7. Rerank the merged shortlist with a cross-encoder, keep the top 5-8 chunks.
8. Pass those chunks, plus the original question, to the LLM with a grounded-answer prompt.

Every stage in that pipeline can be tuned independently. The embedding step is one of eight, and while it matters, chunking and reranking often move retrieval quality further than swapping embedding models does.

FAQ

Do I need a vector database to use embeddings?

No. For corpora up to a few million chunks, Postgres with the pgvector extension handles vector storage and HNSW indexing well, and lets you keep vectors, metadata, and application data in one place. Dedicated vector databases (Pinecone, Weaviate, Qdrant) become worth their operational cost at hundreds of millions of vectors, or when you need advanced features like namespaces per tenant, hybrid search built in, or fully managed scaling. For most mid-market RAG builds, starting with pgvector and migrating later if scale demands it is the pragmatic path.

How much does embedding a large document corpus cost?

Cheaper than most teams expect. OpenAI's text-embedding-3-small is priced at $0.02 per million input tokens. A corpus of one million documents averaging 500 tokens each embeds for about $10. Even text-embedding-3-large at $0.13 per million tokens keeps a million-document corpus under $70. The recurring cost is re-embedding when documents change, which for most organisations is a small fraction of the corpus per month. Self-hosting an open-source model like BGE-M3 removes API costs entirely at the price of running the inference infrastructure.

Can I mix embeddings from different models?

No. Vectors from different models live in different vector spaces and are not comparable. If you embed half your corpus with OpenAI and half with Cohere, cosine similarity between them is meaningless. This is why the choice of embedding model is a longer-term commitment than the choice of LLM: swapping the LLM behind your RAG system takes an afternoon, but swapping the embedding model means re-embedding your entire corpus and re-indexing. Evaluate carefully before you commit at scale.

How is an embedding different from a token?

Tokens are the units of text the LLM reads - usually sub-words like "embed", "##ding", "s". Every token has its own learned vector inside the model, but those are internal and typically not exposed. An embedding in the RAG sense is a single vector that represents an entire chunk of text (a sentence, paragraph, or document), produced by an embedding model designed specifically for that job. Token vectors describe individual word-parts; text embeddings describe whole passages of meaning, and it is the latter you compare with cosine similarity during retrieval.

Do embeddings handle non-English content?

Modern embedding models are increasingly multilingual. Cohere's Embed v4, OpenAI's text-embedding-3-*, and open-source models like BGE-M3 handle 100+ languages and can retrieve across language boundaries - a French query can retrieve a relevant English document if both are embedded by the same multilingual model. Quality varies by language: performance on major European and Asian languages is close to English, while lower-resource languages lag. For UK organisations with multilingual customer bases, a multilingual model is usually the right default, evaluated against a sample of real queries in each language.

How do I know if my embeddings are actually working?

Build an evaluation set of 50-200 real queries paired with the documents that should be retrieved for each, then measure recall@k (does the correct document appear in the top k retrieved chunks). Recall@5 above 80% is a reasonable target for a first cut; above 90% is production-grade. Do this before you tune anything else - if retrieval recall is low, no amount of prompt engineering on the LLM will fix the downstream answer quality. Ragas and TruLens are the two most credible open-source evaluation frameworks for this.

Are embeddings a GDPR concern?

They can be. An embedding is a numerical representation of text, but if the source text contained personal data, the embedding is derived from personal data and remains within scope of UK GDPR. Research has shown that embeddings can be partially inverted to reconstruct source text, so treating them as anonymised is risky. Practical implications: apply the same access controls, retention policies, and processor agreements to your vector store as you would to the source documents. If you are sending personal data to a third-party embedding API, that transfer needs a lawful basis and a DPA in place, per the ICO's AI guidance.

When should I fine-tune an embedding model?

When retrieval quality on your specific domain plateaus below what you need, and you have collected a few thousand real query-document pairs from your system. Fine-tuning a general model on domain-specific pairs typically lifts recall by 5-15 percentage points. It is more effort than swapping models but less than most teams assume - libraries like Sentence Transformers make it a two-day job for someone comfortable with Python. Do it after you have exhausted the cheaper wins: better chunking, hybrid retrieval, reranking, synthetic questions. Fine-tuning last, not first.

Where this fits in a build

Embeddings are one component of a RAG system, and the interesting engineering work almost always sits around them rather than in them. Chunking strategy, hybrid retrieval, reranking, evaluation harnesses, and the ingestion pipeline that keeps everything fresh are where production quality is won or lost. If you are scoping a RAG build and the vendor conversation spends more time on which embedding model than on those surrounding decisions, that is a signal worth acting on.

At AI Advisory we build RAG systems for UK mid-market clients across legal, financial services, and B2B SaaS, with an evaluation-led approach that measures retrieval quality before answer quality. If you would like to talk through a specific use case or review an existing RAG system that is underperforming, get in touch.

Ready to put this into production? book a discovery call.