RAG vs Fine-Tuning: How to Choose Between Retrieval and Model Training
A practical guide to choosing between RAG and fine-tuning for LLM applications: cost, accuracy, latency, maintenance, and when to combine both
Every team building on top of large language models eventually hits the same fork in the road. The base model does not know your product, your policies, or your customers' order history. You have two credible ways to fix that: retrieval-augmented generation (RAG), where you fetch relevant context at query time and inject it into the prompt, or fine-tuning, where you continue training the model on your own data so the knowledge or behaviour becomes part of the weights.
The internet is full of hot takes claiming one always beats the other. In production the answer is almost always "it depends, and often both." This guide is the decision framework we use with clients when scoping RAG pipelines and custom AI solutions - what each approach is actually good at, what it costs, where it breaks, and how to combine them.
What RAG and fine-tuning actually do
The two techniques solve different problems, and confusing them is the root of most bad architecture decisions.
RAG keeps the model frozen and changes what you feed it. At query time, a retriever - typically a vector search over embeddings, often combined with keyword search (BM25) in a hybrid setup - pulls the most relevant chunks from your knowledge base. Those chunks get stitched into the prompt as context, and the model generates an answer grounded in what it just read. The model's weights never change. If you update a policy document tonight, tomorrow's answers reflect it.
Fine-tuning changes the model itself. You take a base model (say, GPT-4o-mini, Llama 3.1, or Mistral) and continue training it on curated examples. The model updates its parameters so that patterns from your data become part of how it responds. This is how you teach a model a specific tone of voice, a structured output format it kept getting wrong, or a domain-specific reasoning style. Modern fine-tuning is usually parameter-efficient - LoRA or QLoRA adapters rather than full retraining - which brings costs down substantially but does not change the fundamental trade-off.
The clearest way to hold the distinction in your head, borrowed from a widely-cited framing by Andrej Karpathy and echoed in OpenAI's own guidance: RAG is for knowledge; fine-tuning is for behaviour. Facts, documents, records, current state - retrieve. Format, style, tone, task-specific reasoning - fine-tune.
When RAG is the right default
For roughly 80% of enterprise LLM applications we scope, RAG is the correct starting point. The reasons are practical rather than ideological.
Your knowledge changes. If your source data updates weekly, daily, or in real time - product catalogues, support articles, contract terms, pricing, inventory, CRM records - fine-tuning is the wrong tool. Retraining a model every time a document changes is absurd; retrieval handles it for free. A customer support assistant grounded in Zendesk articles should read the current article, not a snapshot from three months ago frozen into weights.
You need citations. Regulated industries - financial services, healthcare, legal, insurance - typically need answers to point back to source documents. RAG gives you this natively because you know exactly which chunks the model saw. Fine-tuned models cannot cite their training data at inference time; they just generate. For UK organisations working under FCA guidance, ICO expectations on automated decision-making, or clinical governance, source-linked answers are effectively non-negotiable.
You want to control hallucinations. A well-built RAG system with proper refusal patterns will say "I don't have information on that" when the retrieval returns nothing relevant. Fine-tuned models tend to confabulate confidently on gaps, because generation is their only mode.
You have volume in the knowledge base but not in labelled examples. Fine-tuning needs hundreds to thousands of high-quality input-output pairs. Most organisations have thousands of documents but only a handful of curated Q/A examples. RAG uses the documents directly.
Costs are predictable and mostly linear. You pay for embeddings (one-off per chunk plus updates), vector storage, and inference tokens. There is no training run to blow the budget on.
When fine-tuning earns its place
Fine-tuning is not obsolete - it is just narrower than most vendors selling it will admit. The cases where it genuinely wins:
Structured output the base model keeps getting wrong. If you need JSON in a very specific schema, XML in a legacy format, or SQL against an unusual dialect, fine-tuning a small model on a few thousand examples often outperforms prompting a large model. This is where the cost story flips: a fine-tuned 8B model running on your own infrastructure can be an order of magnitude cheaper per call than GPT-4-class API usage, and faster.
Tone, voice, and stylistic consistency. A well-known example from OpenAI's cookbook covers fine-tuning for brand voice - the model learns how your organisation writes, which is hard to fully specify in a system prompt. Marketing copy generation, response drafting for high-volume support teams, and internal documentation summarisation are all good candidates.
Task-specific reasoning patterns. Classification, entity extraction, intent detection, and other narrow tasks with clear ground truth benefit from fine-tuning. You are not teaching the model new facts; you are teaching it to do one thing very well and cheaply.
Latency and cost at scale. If you are running millions of calls per month, a fine-tuned small model on your own GPUs or via a serverless endpoint (Fireworks, Together, Modal) can shave real money off the bill. For low-volume applications this argument does not hold - the operational overhead outweighs the savings.
Privacy or air-gapped deployment. If regulation or contract prevents data leaving your infrastructure, fine-tuning an open-weights model (Llama 3, Mistral, Qwen) that you host yourself may be the only option. You still often want RAG on top - but the base capability has to live on-prem.
The hybrid approach, which is what actually ships
In production, the interesting systems combine both. A useful mental model:
- Fine-tune the behaviour - format, tone, refusal patterns, tool-use style.
- Retrieve the knowledge - documents, records, current state.
A concrete example. A financial services client of ours wanted an internal research assistant that would answer analyst questions grounded in their proprietary research library, but respond in the house style used in client-facing memos. RAG alone gave accurate answers in a generic voice. Fine-tuning alone gave beautifully written but occasionally wrong answers. The shipped system fine-tuned a mid-sized model on 2,000 curated memo excerpts to lock in voice and structure, then wrapped it in a hybrid retrieval pipeline (BM25 + dense embeddings via pgvector, reranked with a cross-encoder) over the research library. Accuracy on their eval set went from 71% (RAG only) to 88% (hybrid), and analysts stopped complaining that the answers "didn't sound like us."
The order matters: start with RAG, measure honestly, and only reach for fine-tuning when a specific failure mode - not general dissatisfaction - points at behaviour rather than knowledge.
Cost, timeline, and operational reality
The economics are less dramatic than either camp suggests. Rough numbers from projects we have scoped in the last 18 months:
RAG build. A production-grade RAG system for a mid-market client typically runs 8-14 weeks from kickoff to launch. Costs cluster in three buckets: engineering (the largest), embeddings and vector store (small - even large corpora rarely exceed a few hundred pounds a month on managed pgvector or Pinecone), and inference. Ongoing operation is mostly inference tokens plus modest re-embedding as documents change. Total first-year cost for a mid-market deployment: typically £40k-£120k build, £1k-£8k monthly run rate depending on volume.
Fine-tuning project. If you already have clean training data, a fine-tuning run itself is cheap - often under £1,000 for a LoRA adapter on a mid-sized open model via a managed service, or a few hundred dollars via OpenAI's fine-tuning API. The expensive part is the data: curating, labelling, and evaluating 1,000-10,000 high-quality examples typically eats 60-80% of the project budget. Expect £25k-£80k for a first fine-tuning project done properly, plus infrastructure for serving if you are self-hosting.
Hidden costs people forget. Evaluation harnesses (you need one; "looks good in the demo" is not a launch criterion). Monitoring for retrieval quality drift and answer quality drift. Re-embedding when you change embedding models. Version control for prompts, adapters, and index snapshots. Human-in-the-loop review for at least the first 90 days of production traffic. These apply to both approaches and are where under-scoped projects die.
A decision framework you can actually use
Work through these questions in order. Stop at the first clear answer.
- Does the knowledge change more than monthly? If yes, start with RAG. Fine-tuning stale data is a maintenance nightmare.
- Do you need to cite sources or show provenance? If yes, RAG. Non-negotiable in most regulated contexts.
- Is the problem primarily about format, tone, or narrow task specialisation with stable inputs? If yes, consider fine-tuning - possibly without RAG at all.
- Do you have 1,000+ high-quality labelled examples, or budget to create them? If no, fine-tuning is off the table for now. RAG or better prompting.
- Are you running enough volume that inference cost is a real line item (roughly £5k+/month)? If yes, fine-tuning a smaller model becomes economically interesting. If no, stay on frontier APIs.
- Do you have both a knowledge problem AND a behaviour problem? Plan for hybrid, but ship RAG first. Add fine-tuning in phase two once you have production data and clear failure modes.
One anti-pattern worth calling out: teams who cannot get RAG to work well and reach for fine-tuning as a fix. Fine-tuning does not fix bad retrieval. If your chunks are wrong, your metadata is thin, or your reranking is absent, the model will still hallucinate - it will just do it in your house style. Fix the retrieval pipeline first.
What to build first if you are starting today
For most teams, the pragmatic sequence looks like this. Ship a RAG prototype in weeks two to four using off-the-shelf components - pgvector or Pinecone for retrieval, a frontier model API for generation, a small evaluation set of 50-100 realistic queries with expected answers. Instrument everything. Get it in front of real users, even a small internal group, and log every query, retrieval result, and generation.
Only after four to eight weeks of production traffic do you have enough signal to make the RAG-vs-fine-tuning-vs-hybrid call properly. You will know which queries fail, whether the failures are knowledge (fix retrieval) or behaviour (consider fine-tuning), and what the actual usage volume and cost profile looks like. Deciding earlier is guessing.
If you are weighing this decision for a real project and want a sanity check on the architecture, the data requirements, or the build vs buy calculus, get in touch with AI Advisory - we scope RAG and fine-tuning projects most weeks, and a 30-minute conversation usually saves months of building the wrong thing.
Frequently asked questions
Is RAG cheaper than fine-tuning?
Usually yes, at least initially. RAG has no training cost and lower data-preparation overhead - you index documents rather than curate labelled examples. A production RAG build typically runs £40k-£120k for mid-market scope, versus £25k-£80k for a first fine-tuning project where 60-80% of the budget goes on data curation and evaluation. However, fine-tuning can flip the economics at scale: if you run millions of calls per month, a fine-tuned small model on your own infrastructure can cut per-call inference costs by 5-10x compared to frontier model APIs, which more than pays back the upfront investment within 6-12 months.
Can fine-tuning teach a model new facts reliably?
Poorly, and this is one of the most common misconceptions. Fine-tuning updates model weights based on patterns in your training data, but the model does not "memorise" facts in a queryable way. It may reproduce trained facts sometimes and hallucinate confidently at other times, with no clear signal to the user which is happening. Research from Meta, Google DeepMind, and others has consistently shown that RAG outperforms fine-tuning for factual recall tasks. If your goal is to inject knowledge - product details, policies, records - use retrieval. Fine-tuning is for behaviour, format, and style.
How much data do I need for fine-tuning to be worthwhile?
The rough floor is 500-1,000 high-quality examples for narrow tasks like classification or format enforcement. For richer tasks - tone, multi-step reasoning, complex output structures - you typically want 2,000-10,000 examples. Quality matters more than quantity: 500 carefully curated, human-reviewed examples will usually beat 5,000 noisy ones. If you have fewer than 500 examples and no realistic path to creating more, fine-tuning is not the right tool yet. Focus on prompt engineering, few-shot examples, and RAG instead. You can revisit fine-tuning once production traffic gives you real data to learn from.
Does RAG work with sensitive or regulated data under UK GDPR?
Yes, and it is often the safer choice from a compliance standpoint. Because RAG keeps your data in your own vector store and only sends relevant chunks to the model at query time, you retain granular control over what gets processed and where. You can host embeddings and retrieval infrastructure in-region (UK or EU), apply row-level access controls so users only see documents they are authorised for, and log exactly what was retrieved for each query - useful for ICO audit requirements. If you cannot send data to third-party model APIs at all, pair RAG with a self-hosted open-weights model (Llama 3.1, Mistral) to keep the entire pipeline within your infrastructure.
Can I fine-tune on top of a RAG system to get the best of both?
Yes, and this is the most common architecture in mature production deployments. The pattern is straightforward: fine-tune a base model on your desired behaviour (tone, output format, refusal patterns, tool-use style) using a few thousand curated examples, then deploy that fine-tuned model as the generator inside a RAG pipeline. Retrieval handles current knowledge; the fine-tuned weights handle how the model responds. This gives you the freshness and citation trail of RAG with the consistency and cost profile of a specialised model. Do not build this on day one - ship RAG first, learn the failure modes, then add fine-tuning where it earns its place.
What breaks in production, and who maintains it?
For RAG: retrieval quality drift (new document types the chunking strategy handles badly), embedding model updates (which require re-indexing), and prompt regression when you change the generator. For fine-tuning: distribution shift (the real inputs diverge from your training data), base model deprecation (OpenAI retires the model you fine-tuned on), and silent accuracy degradation that only shows up in user complaints. Both need an evaluation harness running against a golden set on every change, monitoring dashboards for answer quality and latency, and a named owner - typically an ML or platform engineer with 0.2-0.5 FTE allocated. Under-resourcing operations is where most enterprise LLM projects quietly fail.
How long does a first build take end-to-end?
For a well-scoped RAG deployment, expect 8-14 weeks from kickoff to production, with a working prototype in weeks two to four. The first two weeks are discovery, source data audit, and evaluation set definition. Weeks three to eight are pipeline build, iteration on chunking and retrieval, and refusal pattern tuning. Weeks nine to fourteen cover integration, security review, and staged rollout. A first fine-tuning project is often quicker in calendar time (6-10 weeks) but front-loaded with data work: expect 4-6 weeks curating and labelling examples before any training run. Hybrid projects typically add 4-6 weeks on top of RAG timelines.
What if I have already tried RAG and it did not work?
Most RAG failures are retrieval failures dressed up as generation failures. Before concluding you need fine-tuning, audit the retrieval layer honestly: are you using hybrid search (dense + BM25), or dense embeddings alone? Is there a reranker? Are chunks the right size for your documents - typically 300-800 tokens with overlap? Is metadata filtering being used to narrow the search space? Are you evaluating retrieval separately from generation, so you know whether the model got the right context in the first place? In our experience, roughly 70% of "RAG doesn't work" cases are fixed by improving retrieval, not by switching to fine-tuning. Diagnose the actual failure mode before changing architecture.
Further reading
Sources referenced for context not directly cited in the body: