What is RAG in Machine Learning? A Practical Explanation

Retrieval-augmented generation, or RAG, is the machine learning pattern that lets a large language model answer questions using information it was never trained on. Instead of relying only on the model's frozen parameters, a RAG system fetches relevant documents from an external store at query time and passes them to the model as context. The model then generates its answer grounded in those retrieved passages.

The pattern was formalised in a 2020 paper from Facebook AI Research (Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"), but it has become the default architecture for enterprise AI assistants because it solves three problems that pure LLMs cannot: stale knowledge, hallucination on facts the model does not know, and the inability to cite sources.

This article explains what RAG actually is at a mechanical level, how the components fit together, when it is the right choice versus fine-tuning, and what tends to break when you put it into production.

The problem RAG solves

A large language model like GPT-4, Claude, or Llama 3 is a fixed set of weights trained on a snapshot of text data. Once training finishes, the model knows what it knew on the cutoff date and nothing more. If you ask it about your company's internal expenses policy, a contract signed last Tuesday, or a product SKU that launched this morning, it has three options: refuse, guess plausibly (hallucinate), or answer from adjacent knowledge that may or may not apply.

You have two ways to give the model new knowledge. You can retrain or fine-tune it on your data, which is expensive, slow, and locks the knowledge into the weights where it cannot be updated without another training run. Or you can hand the model the relevant information at inference time as part of the prompt, and let it read and reason over that information to produce an answer. RAG is the second approach, engineered so it works at scale across millions of documents.

The practical wins are considerable. Knowledge updates become a matter of re-indexing documents rather than retraining. Answers can cite the exact passage they were drawn from, which matters for legal, medical, and financial use cases. Hallucination rates drop substantially because the model has the actual source text in front of it. And access controls stay at the document layer, which is where your existing permissions live.

How a RAG system actually works

A production RAG system has two phases: an offline indexing phase and an online query phase.

Indexing. You take your source documents (PDFs, wiki pages, database rows, tickets, transcripts, whatever) and process them into a searchable form. That means:

1. Chunking - splitting documents into passages small enough to fit inside the model's context window, typically 200-800 tokens each, with some overlap so meaning is not cut mid-sentence.
2. Embedding - passing each chunk through an embedding model (OpenAI's text-embedding-3, Cohere's embed-v3, or open-source options like BGE and E5) to produce a dense vector, typically 768 to 1536 dimensions, that captures the semantic meaning of that chunk.
3. Storing - writing those vectors into a vector database or vector-enabled store: Pinecone, Weaviate, Qdrant, Milvus, or Postgres with the pgvector extension. Alongside each vector you store the original text, the source URL or document ID, and any metadata you want to filter on (owner, date, permission group, product line).

Query. When a user asks a question, the system:

1. Embeds the question using the same embedding model, producing a query vector.
2. Runs a similarity search (cosine similarity, dot product, or Euclidean distance) against the vector store to find the top-k most similar chunks, typically 5 to 20.
3. Optionally re-ranks those chunks with a cross-encoder model like Cohere Rerank or BGE-reranker, which is slower but more accurate than pure vector similarity.
4. Constructs a prompt that includes the retrieved chunks as context, the user's question, and instructions telling the model to answer from the provided context and cite sources.
5. Sends the prompt to the LLM and returns the generated answer, usually with citation links back to the source chunks.

That is the whole loop. Everything else - hybrid search, query rewriting, HyDE, agentic retrieval, GraphRAG - is a variation on or extension of these steps.

Why vector search alone is not enough

Early RAG systems used pure vector similarity, and they worked adequately for demos. In production they fall over on a predictable set of failure modes.

Vector search is strong on semantic similarity but weak on exact-match retrieval. If a user asks about "invoice INV-4471-B" or "SKU 3388-X", a semantic embedding does not particularly care about the string. It will return chunks about invoices in general, not the specific invoice. Keyword search - the sort BM25 does - is much better at this.

Production systems therefore use hybrid retrieval: run both a vector search and a BM25 keyword search, then combine the results using reciprocal rank fusion or a learned weighting. Elastic, OpenSearch, Weaviate, and Qdrant all support this natively. Microsoft's own guidance for Azure AI Search (see Microsoft Learn on hybrid search) treats hybrid + semantic ranker as the default configuration for RAG.

The second common upgrade is query rewriting. Users ask ambiguous questions. "What did we agree with them last quarter?" contains no proper nouns and no dates the retriever can use. A small LLM call to rewrite that into a well-formed search query, given conversation history and any known context, materially improves retrieval quality. HyDE (Hypothetical Document Embeddings) goes further and asks the LLM to imagine what an ideal answer would look like, then embeds that hypothetical answer and searches with it.

The third is re-ranking. Vector search returns candidates fast but imperfectly. A cross-encoder re-ranker scores each candidate against the query jointly, which is slower per pair but far more accurate. Retrieving 50 candidates and re-ranking down to 8 usually beats retrieving 8 candidates directly.

RAG vs fine-tuning: choosing the right tool

The single most common confusion for teams starting with LLMs is when to use RAG and when to fine-tune. They solve different problems.

Use RAG when the goal is to inject knowledge the model does not have: your product docs, your policies, your codebase, your contract library, your ticket history. Knowledge that changes over time, needs to be cited, has access controls, or is too large to fit in a prompt. This is roughly 80% of enterprise AI use cases.

Use fine-tuning when the goal is to change the model's behaviour, style, or output format: producing SOAP notes in a specific structure, writing in a house voice, classifying tickets into a fixed taxonomy, extracting structured JSON reliably, or teaching the model a task it does badly out of the box. Fine-tuning is about how the model responds, not what facts it knows.

You often want both. A common production pattern is a lightly fine-tuned model that has learned your output format and refusal patterns, sitting on top of a RAG pipeline that supplies the domain knowledge. OpenAI's own guidance (Optimizing LLM Accuracy) is explicit that RAG addresses knowledge gaps and fine-tuning addresses behaviour gaps, and that most teams should exhaust RAG first because it is cheaper, faster to iterate, and easier to audit.

Evaluation: how you know your RAG works

The failure mode that kills most RAG projects is not that the answers are wrong, but that no one can tell whether they are wrong at scale. You need an evaluation harness before you have a production system, not after.

Evaluation splits into two questions. First, is the retriever finding the right chunks? This is measured with retrieval metrics: recall@k (did the correct chunk appear in the top k?), MRR (mean reciprocal rank of the correct chunk), and nDCG (rank-weighted relevance). You need a labelled test set of questions paired with the chunks that ought to answer them. A few hundred labelled questions is enough to start.

Second, is the generator producing a correct, grounded answer given the retrieved context? This is measured with generation metrics: faithfulness (does every claim in the answer follow from the retrieved context?), answer relevance (does the answer address the question?), and context relevance (were the retrieved chunks actually useful?). The Ragas library and DeepEval are the two open-source standards. Both work by using a strong LLM as a judge to score answers along these axes.

Track these metrics over time. Every change to chunking, embedding model, retrieval strategy, or prompt should be evaluated against the same test set. Without this, you are guessing.

The costs and constraints that matter in practice

RAG is not free, and the cost model is different from what most teams expect.

The dominant cost is inference tokens, not embeddings or storage. Embedding a million chunks with OpenAI's text-embedding-3-small costs roughly £15-£20 at current pricing. Storing those vectors in pgvector or Qdrant is essentially free at that scale. But every user query sends the retrieved context to the LLM, and if you retrieve 10 chunks of 500 tokens each plus a system prompt, you are paying for 6,000+ input tokens per query. At GPT-4o pricing, that is roughly 1.5p per query. At 100,000 queries a month, that is £1,500 in inference costs alone.

Latency is the second constraint. A full RAG loop - embed query, vector search, re-rank, LLM generation - typically runs 1.5 to 4 seconds end-to-end. Vector search itself is fast (tens of milliseconds); the LLM generation is where the time goes. Streaming the response back to the user hides this reasonably well for chat interfaces but not for automation pipelines that need a complete answer before proceeding.

Data governance is the third. RAG systems must respect the same access controls as the underlying documents. If a sales rep should not see the CFO's board pack, the retriever must filter on user permissions before returning candidates. This is done with metadata filters on the vector store and requires that permissions are indexed alongside each chunk. Getting this wrong is a data breach, and the UK ICO has been increasingly explicit that AI systems processing personal data must implement appropriate technical measures (see ICO guidance on AI and data protection).

Where RAG is heading

The pattern is stabilising but not standing still. Three developments matter for teams planning production systems in 2026.

Long-context models are not killing RAG. Gemini 1.5 Pro and Claude with 200k+ context windows raised the question of whether you can just stuff all your documents into the prompt. In practice, no: cost scales linearly with input tokens, latency degrades, and retrieval accuracy from within a long context is worse than targeted retrieval (documented in the "lost in the middle" literature, e.g. Liu et al., 2023). Long context helps RAG by letting you include more retrieved chunks; it does not replace retrieval.

Agentic RAG extends the basic loop with iterative retrieval. Instead of retrieving once and generating, the model decides what to search for, evaluates whether the results are sufficient, and searches again if not. This handles multi-hop questions ("which of our contracts signed in 2024 have auto-renewal clauses and expire before June?") that single-shot retrieval cannot answer well.

GraphRAG, popularised by Microsoft Research, builds a knowledge graph from the source corpus during indexing and uses graph traversal alongside vector search at query time. It excels at questions requiring reasoning across relationships ("how does policy X affect team Y given constraint Z?") but is heavier to build and maintain than a standard vector-only pipeline.

For most mid-market teams, a well-built hybrid-search RAG system with proper evaluation will outperform an under-engineered agentic or graph-based system. Get the fundamentals right first.

Frequently asked questions

Is RAG a form of machine learning or just an engineering pattern?

Both. The individual components - the embedding model, the LLM, and often the re-ranker - are all machine learning models, typically transformers trained on large text corpora. What RAG contributes is the orchestration pattern that combines them with a retrieval system at inference time. You do not train a "RAG model" as a single thing; you compose existing models with a retrieval layer. That is why RAG is often described as an architecture rather than a model. It sits at the boundary between ML and information retrieval, borrowing techniques from both fields.

What is the difference between RAG and semantic search?

Semantic search returns a ranked list of documents given a query. RAG uses semantic search (usually as one component) and then feeds the results to a language model that generates a natural-language answer grounded in those documents. If you stopped at the ranked list, you would have semantic search. If you added an LLM that reads the results and writes a synthesised answer with citations, you would have RAG. Most production RAG systems also add hybrid keyword search, re-ranking, query rewriting, and access-control filtering that pure semantic search implementations do not need.

How much data do I need to make RAG useful?

There is no minimum, but the value scales with corpus size and query volume. If you have 20 documents that fit in a prompt, you do not need RAG - just paste them into context. RAG becomes valuable somewhere between a few hundred and a few thousand documents, where you cannot fit them all in context but the knowledge is finite enough to index. It becomes essential above roughly 10,000 documents or whenever documents change frequently. The other trigger is citation: even with a small corpus, if you need to point users at the specific source of every claim, RAG is the right pattern.

Can RAG hallucinate?

Yes, though less than a bare LLM. The model can still ignore the retrieved context, blend it with its training-data priors, or fabricate citations. The mitigations are strong system prompts ("answer only from the provided context; if the answer is not present, say so"), grounded generation techniques that force citations at the token level, and faithfulness evaluation to catch drift. In legal, medical, and financial applications, you also add a verification step that checks each factual claim against the source chunks before returning the answer. Expect a residual hallucination rate of 1-5% in well-built systems and design your UX around that.

Do I need a vector database, or will Postgres do?

For most projects up to roughly 10 million vectors, Postgres with the pgvector extension is entirely adequate and often the right choice. You get transactional consistency, familiar operations, existing backup procedures, and the ability to filter on any column. Dedicated vector databases (Pinecone, Weaviate, Qdrant, Milvus) become worth their operational overhead when you need very high query throughput, distributed indexing across many nodes, or advanced indexing options like filtered HNSW at scale. Start with pgvector; migrate when you have a measured reason to.

How does RAG handle documents that update frequently?

You re-index changed documents. In practice this means running a pipeline that watches your source system (a wiki, a CMS, a shared drive), detects changes, re-chunks and re-embeds affected passages, and upserts them into the vector store. For most systems this runs on a schedule (every hour or every night) or in response to webhook events. The cost of re-embedding is small compared to inference, and vector stores handle upserts efficiently. The engineering effort is in the change-detection layer, particularly for document formats like PDFs where a small edit can shift chunk boundaries and force re-indexing of the whole document.

How long does it take to build a production RAG system?

A functional prototype takes days. A production system with proper evaluation, access controls, monitoring, and edge-case handling typically takes 8-16 weeks for a mid-market implementation, depending on how messy the source data is. The build itself is usually not the bottleneck; getting clean, well-permissioned access to the source documents is. Budget more time than you think for data engineering, chunking strategy for your specific document types (contracts and technical manuals need different approaches), and building an evaluation set with domain experts. Skipping evaluation to ship faster is the single most common reason RAG projects underperform in production.

Is RAG suitable for regulated industries under UK GDPR?

Yes, and it is often easier to make compliant than fine-tuning because the knowledge stays in a controllable data store rather than being baked into model weights. The requirements are the same as for any system processing personal data: lawful basis, data minimisation in what you index, access controls on retrieval, retention policies on the vector store, and appropriate technical measures around the LLM provider (typically an enterprise contract with data residency and no-training guarantees). The ICO's AI guidance is the authoritative reference. Self-hosted embedding and generation models are worth considering for highly sensitive data where sending text to a third-party API is not acceptable.

Next steps

RAG is now the default architecture for enterprise AI assistants because it fits how real organisations actually work: knowledge lives in existing document stores, permissions matter, sources need citing, and information changes daily. The pattern is well-understood, the tooling is mature, and the failure modes are known. What separates production systems from demos is discipline around retrieval quality, evaluation, and governance - not novelty in the architecture.

If you are scoping a RAG build and want a pragmatic view of what to prioritise, where the costs actually land, and how to evaluate whether it is working, AI Advisory runs discovery engagements that produce a costed, buildable specification rather than another deck.

Ready to put this into production? book a discovery call.