AI26 July 20265 min read

RAG in AI Explained in Simple Terms

RAG explained in plain English: what retrieval-augmented generation is, how it works, when to use it, and how it compares to fine-tuning

By AI Advisory team

Retrieval-Augmented Generation, or RAG, is the technique that lets a large language model answer questions using your documents instead of only what it learned during training. In plain terms: before the model writes an answer, a retrieval step fetches the most relevant snippets from a knowledge base you control, and those snippets are pasted into the prompt as context. The model then answers based on that context.

That is the whole idea. Everything else - vector databases, embeddings, chunking, re-rankers, hybrid search - is machinery to make the retrieval step accurate and fast. This article walks through what RAG actually does, why teams use it, how a working system fits together, and where it fails.

Why RAG exists at all

Large language models like GPT-4, Claude, and Llama are trained on a fixed snapshot of public text. That snapshot has three problems for business use.

First, it is out of date the moment training finishes. A model trained with data up to early 2024 does not know about a policy change your company made in June. Second, it never contained your private data in the first place - your contracts, your internal wiki, your support tickets, your product specs. Third, when a model does not know something, it often invents a plausible answer rather than admitting the gap. This is what people mean by hallucination.

You could solve this by retraining the model on your data every time something changes. That is expensive, slow, and technically unnecessary for most cases. RAG solves the same problem by keeping the model as-is and instead giving it the right reference material at the moment of the question - the same way a lawyer answers a client question by pulling the relevant case file rather than reciting from memory.

The original RAG paper from Meta AI in 2020 introduced the pattern formally, but the underlying idea - retrieve, then generate - is now the default architecture for almost every serious enterprise AI assistant.

How RAG works, step by step

A working RAG system has two phases: an indexing phase that happens once (and then on updates), and a query phase that runs every time a user asks a question.

Indexing phase

You start with a pile of source documents - PDFs, Word files, wiki pages, database records, transcripts, whatever contains the knowledge you want the AI to use. Then:

1. Chunking. Documents are split into smaller pieces, usually 200 to 800 words each. You cannot dump a 300-page manual into a prompt, and even if you could, retrieval works better on focused chunks.

2. Embedding. Each chunk is passed through an embedding model - OpenAI's text-embedding-3, Cohere Embed, or an open-source model like BGE - which turns the text into a list of numbers (a vector) of typically 768 to 3072 dimensions. Chunks with similar meaning end up with similar vectors.

3. Storage. The vectors are stored in a vector database - Pinecone, Weaviate, Qdrant, or Postgres with the pgvector extension. Each vector is stored alongside the original chunk text and metadata like source, date, and permissions.

Query phase

When a user asks a question:

1. Embed the question. The same embedding model turns the user's question into a vector.

2. Search. The vector database finds the top N chunks (typically 5 to 20) whose vectors are closest to the question vector. Closeness here means semantic similarity, not keyword matching - a question about "annual leave" will find chunks about "holiday entitlement" even without shared words.

3. Assemble the prompt. The retrieved chunks are pasted into a prompt template that looks roughly like: "Answer the user's question using only the context below. If the context does not contain the answer, say so. Context: [chunks]. Question: [user question]."

4. Generate. The LLM produces an answer grounded in the supplied context, ideally with citations pointing back to the source documents.

That is the entire loop. A well-built RAG system does this in under two seconds end to end.

A simple worked example

Imagine a professional services firm with 12 years of client engagement documents, roughly 40,000 files. Consultants routinely ask questions like "have we ever worked with a client in the pharmaceutical sector on a supply chain transformation, and if so, what was the approach?"

Without RAG, the model would either guess or refuse. With RAG:

All 40,000 documents are chunked into roughly 600,000 chunks and embedded into a vector database. This is a one-time job that might take a few hours and cost £200-£400 in embedding API calls.
The consultant's question is embedded and searched against the index. The top 15 chunks come back: three from a 2022 GSK engagement, four from a 2020 AstraZeneca proposal, and eight from methodology documents.
Those chunks are handed to the LLM along with the question. The model produces a two-paragraph answer citing the specific engagements and pointing to the source files.

The consultant gets the answer in seconds instead of spending 40 minutes searching SharePoint. The firm's data never leaves its infrastructure if the system is self-hosted. And crucially, the answer is grounded - if the retrieval step returns nothing relevant, the model says so rather than inventing a client engagement that never happened.

Why not just fine-tune the model?

This is the most common question when people first hear about RAG. Fine-tuning means further training a base model on your specific data so it learns your content. It works, and there are situations where it beats RAG. But for most business cases, RAG wins on four counts.

Freshness. When a document changes, you re-embed one chunk. With a fine-tuned model, you retrain, which takes hours to days and costs meaningfully more.

Auditability. RAG can cite the exact source of each claim. A fine-tuned model produces answers with no traceable provenance, which is a problem for regulated sectors and for anyone who cares about verification.

Permissions. RAG can filter retrieval by user identity - a junior consultant sees different chunks from a partner. Fine-tuned models bake all training data into their weights with no way to enforce per-user access.

Cost. Fine-tuning a capable model costs thousands to tens of thousands of pounds per run and locks you into that base model. RAG works with any LLM you can call via API and adds only the cost of embeddings and vector storage, which is trivial by comparison.

Fine-tuning earns its keep when you need the model to adopt a specific style, format, or reasoning pattern - not when you need it to know facts. In practice, the strongest systems often combine both: a lightly fine-tuned model that speaks in the company's voice, plus RAG for the factual grounding.

Where RAG breaks, and how good systems handle it

Basic RAG - naive chunking, single embedding search, no re-ranking - works for demos and falls over in production. The failure modes are predictable.

Retrieval misses the right chunk. Vector search is fuzzy. If the question uses different vocabulary from the source, the relevant chunk might not make the top 15. Fix: hybrid search that combines vector similarity with traditional keyword search (BM25), then re-ranks the combined results with a cross-encoder model like Cohere Rerank.

Chunks lack context. A chunk saying "the deadline is 30 days" is useless without knowing what deadline. Fix: include document titles, section headers, and neighbouring context in each chunk, or use contextual chunking approaches where a small LLM adds a one-sentence summary of the parent document to each chunk before embedding.

Questions require reasoning across many documents. "Which clients have missed SLAs more than twice this year?" cannot be answered by retrieving 15 chunks. Fix: agentic RAG or query decomposition, where the model first breaks the question into sub-queries, runs multiple retrievals, and reasons over structured results. Or - simpler and often better - send the question to a SQL query against structured data instead.

The model still hallucinates. Even with good context, LLMs sometimes fabricate. Fix: explicit refusal patterns in the prompt ("if the context does not contain the answer, respond with 'I don't know based on the available documents'"), plus an evaluation harness that regularly tests the system against a set of known-answer questions and flags regressions.

Stale index. Documents change; the index does not. Fix: an ingestion pipeline that watches source systems (SharePoint, Google Drive, Notion, your CMS) and re-embeds changed content automatically. This is where workflow automation tools like n8n earn their keep in a RAG stack.

What a RAG project actually costs and how long it takes

For a mid-market business commissioning a first RAG system - internal knowledge assistant, customer support copilot, or similar - realistic ranges look like this.

Proof of concept: £8,000-£20,000, 3-6 weeks. Single document source, 1,000-10,000 documents, one user interface, basic evaluation. Enough to prove value with a specific team.

Production system: £40,000-£120,000, 8-16 weeks. Multiple source systems with live ingestion, hybrid retrieval with re-ranking, evaluation harness, permissions model, monitoring and logging, and a supported user interface (usually Slack, Teams, or a web app).

Ongoing operation: £2,000-£8,000 per month. Covers infrastructure (LLM API calls, vector database, hosting), monitoring, evaluation runs, and iteration based on user feedback. Larger deployments with millions of documents or high query volume push this higher.

The main variables are document volume, the number of source systems you need to ingest from, and how strict the permissions and audit requirements are. A regulated financial services deployment with per-user access control and full audit logging costs materially more than an internal marketing knowledge base.

UK-specific considerations: if your documents contain personal data, the ICO's guidance on AI and data protection applies, and you will want to think carefully about where embeddings and LLM inference happen. Self-hosted embedding models and EU-region LLM endpoints (Azure OpenAI in UK South, AWS Bedrock in London) are the usual answers.

When RAG is the right tool - and when it isn't

RAG is the right answer when:

You have a body of text-based knowledge that changes over time
Users ask natural-language questions where the answer lives inside that knowledge
Answers need to be traceable to sources
Different users have different access rights to the underlying content

RAG is the wrong answer when:

The question requires computation or aggregation over structured data (use SQL, or a SQL-generating agent)
The answer requires up-to-the-second data (use a direct API call)
You need the model to adopt a specific style or format regardless of content (fine-tune)
The knowledge fits comfortably in a single prompt (just paste it in - RAG is overkill under a few dozen pages)

The pattern most enterprise AI teams settle on is a router: the user's question hits a small classifier that decides whether to answer from RAG, run a SQL query, call an external API, or refuse. This is more work than a pure RAG demo but is what separates a system that works in production from one that impresses in a boardroom and disappoints six weeks later.

Frequently asked questions

Is RAG the same as ChatGPT with file upload?

Not quite, though the user experience feels similar. When you upload files to ChatGPT or Claude, the platform runs a lightweight RAG pipeline behind the scenes: it chunks, embeds, and retrieves from your uploaded documents. That works for a few files and short-lived sessions. A production RAG system is different in scale and control - it can ingest millions of documents, connect to live sources like SharePoint or Salesforce, enforce per-user permissions, log every query for audit, and be tuned for accuracy on your specific content. Consumer file upload is RAG-as-a-feature; a proper deployment is RAG-as-infrastructure.

Do I need a vector database, or can I use Postgres?

For most mid-market deployments, Postgres with the pgvector extension is entirely sufficient and often the better choice. It handles millions of vectors comfortably, integrates with the database you probably already run, and avoids adding another piece of infrastructure to operate. Dedicated vector databases like Pinecone, Weaviate, and Qdrant earn their place when you need very low latency at scale (hundreds of queries per second), very large indexes (hundreds of millions of vectors), or specific features like advanced filtering and multi-tenancy that are harder to build on Postgres. Start with pgvector; graduate only when you have a concrete reason.

How accurate is RAG in practice?

On well-designed systems with clean source documents, we typically see 85-95% of questions answered correctly and with correct citations, measured against a human-graded evaluation set. The remaining 5-15% split between retrieval failures (right answer exists but was not fetched), reasoning failures (right chunks retrieved but model drew the wrong conclusion), and genuine knowledge gaps (answer not in the corpus). Getting from 70% to 90% accuracy is where most of the engineering effort goes - hybrid retrieval, re-ranking, prompt tuning, and evaluation-driven iteration. Anyone quoting 99% accuracy without showing you the evaluation methodology is selling something.

Can RAG work with images, audio, or video?

Yes, though the machinery differs. For images, multimodal embedding models like OpenAI's CLIP or Cohere's Embed v3 Multimodal can embed images and text into the same vector space, letting you retrieve images by text query or vice versa. For audio and video, the pragmatic approach is to transcribe first (Whisper is the standard) and then run text-based RAG over the transcripts, ideally with timestamps preserved so answers can link back to specific moments. Fully multimodal RAG that reasons over raw video is still an active research area rather than a production-ready pattern for most business cases.

What data protection issues should I think about?

Three main ones under UK GDPR. First, lawful basis for processing personal data through the RAG pipeline - usually legitimate interest for internal systems, but document the assessment. Second, data residency - where do embeddings, indexes, and LLM inference physically happen? UK or EU regions of major cloud providers, or self-hosted, are the low-friction answers. Third, subject access and erasure requests - if someone asks for their data to be deleted, your ingestion pipeline needs to remove the relevant chunks and re-embed. The ICO's AI guidance covers this in more detail, and DPIAs are advisable for any RAG system processing sensitive personal data.

How does RAG compare to giving the model a very long context window?

Modern models like Gemini 1.5 Pro and Claude with 200k-1M token context windows have made "just paste everything in" a viable pattern for smaller corpuses. It is simpler than RAG and often more accurate for the documents that fit. But it breaks down on cost (you pay for every token on every query), latency (long-context queries are slow), and scale (even 1M tokens is only around 750,000 words, roughly 1,500 pages). RAG remains the right pattern when your corpus is larger than the context window, when query volume makes per-query cost matter, or when you need permissions and citation. Some teams now combine both: RAG retrieves a large candidate set, and a long-context model reasons over it.

How long before a RAG system pays for itself?

For internal knowledge assistants, the honest answer is 6-12 months from go-live in most cases, driven by time saved on document search and question-answering. A team of 40 knowledge workers each saving 20 minutes a day is roughly £120,000 of recovered time annually at typical mid-market salary levels, against a £60,000 build and £30,000 annual run cost. Customer-facing deployments (support copilots, sales enablement) can pay back faster because they show up in metrics leadership already watches - ticket deflection, first-response time, conversion rate. The systems that fail to pay back are almost always the ones that skipped evaluation and monitoring, so accuracy drifted and adoption stalled.

Can we build a RAG system in-house or do we need an agency?

A capable engineering team with LLM experience can build a working RAG prototype in a few weeks. The gap between prototype and production system is where most in-house efforts stall - hybrid retrieval, evaluation harnesses, ingestion pipelines, permissions, monitoring, and the operational discipline to keep accuracy from decaying over time. If you have a senior AI engineer plus DevOps capacity to spare for six months, in-house is realistic. If not, the pragmatic path is a specialist build partner for the initial system and either an internal handover or a retained operating relationship afterwards. The economics rarely favour building everything from scratch when the patterns are now well understood.

Getting started

The best first RAG project is small, specific, and measurable: one team, one document source, one clearly defined set of questions users actually ask today. Prove the value there, instrument the accuracy, and expand from that base. Grand plans to "RAG the entire company" almost always underdeliver; targeted deployments almost always pay back and create the appetite for the next one.

If you are weighing up a RAG build for your organisation and want a straight conversation about what is realistic on your data and timeline, AI Advisory runs a two-week discovery that ends with a costed build plan you can take to anyone.