Fine-Tuning vs RAG: What Each Actually Does and When to Use Which

Fine-tuning and retrieval-augmented generation (RAG) are the two techniques people reach for when a base LLM is not good enough on its own. They solve different problems, cost different amounts, and fail in different ways. Most teams confuse them, pick the wrong one, and then blame the model.

This article explains what each technique actually does under the hood, when to pick one over the other, when to combine them, and roughly what each costs to build and run. It is written for technical decision-makers - CTOs, heads of engineering, ML leads - who are scoping a build and want to make the right architectural call before spending money.

What fine-tuning actually is

Fine-tuning takes a pre-trained model and continues training it on your own examples so that its weights shift towards your desired behaviour. You are not teaching the model new facts in any reliable sense. You are teaching it a pattern: a tone, a format, a style of reasoning, a way of classifying inputs, or a domain-specific vocabulary.

There are a few flavours in common use. Full fine-tuning updates every weight in the model and is expensive and rarely necessary. Parameter-efficient methods like LoRA and QLoRA update a small adapter layer, are much cheaper, and are what most teams actually use. Supervised fine-tuning (SFT) is the standard approach - you provide prompt/response pairs. Preference-based approaches like DPO or RLHF are used when you want the model to prefer one style of answer over another rather than learn a new task.

OpenAI, Anthropic, Google, and the open-source ecosystem (Llama, Mistral, Qwen) all offer fine-tuning paths. OpenAI's fine-tuning API supports GPT-4o and GPT-4o-mini with SFT and DPO; the mechanics and pricing are documented in their fine-tuning guide. For open-weight models, Hugging Face's TRL library is the de facto tooling.

The critical point: fine-tuning bakes behaviour into the weights. Once the training run is done, the model behaves that way by default, with no extra context needed at inference time. That is its main strength and its main weakness.

What RAG actually is

Retrieval-augmented generation keeps the model unchanged and instead injects relevant information into the prompt at query time. When a user asks a question, a retrieval system searches a knowledge base - typically a vector store, sometimes combined with keyword search - pulls back the most relevant chunks, and passes them to the LLM as context. The model then generates an answer grounded in those chunks.

A production RAG system has more moving parts than most people expect. You need a chunking strategy (how documents are split), an embedding model (how chunks are represented as vectors), a vector database (Postgres with pgvector, Pinecone, Weaviate, Qdrant), a retrieval strategy (dense-only, hybrid with BM25, reranking with a cross-encoder), a prompt template that presents the retrieved context to the LLM, and an evaluation harness that measures whether retrieval is actually surfacing the right material.

The original RAG paper from Lewis et al. (2020) established the architecture; the field has moved on considerably since, but the core idea holds. What has changed is the sophistication of retrieval - hybrid search, reranking, query rewriting, and agentic retrieval loops are now standard in production systems.

The critical point: RAG keeps knowledge outside the model. You can update the knowledge base at any time without retraining anything. The model reads the source material fresh on every query.

The key differences that decide which to use

The distinction that matters most is knowledge vs behaviour. If you need the model to know things - policies, product details, case history, technical documentation - use RAG. If you need the model to behave a certain way - classify tickets into your taxonomy, write in your house style, produce output in a specific JSON schema, reason through a domain-specific workflow - fine-tuning is a stronger fit.

Freshness is the second axis. RAG handles changing information natively; you re-index and the system reflects the new state. Fine-tuning freezes what the model knows at training time, and updating that knowledge means another training run.

Auditability matters more than most teams realise until they hit compliance. RAG systems can cite the exact source chunk used to answer a question, which is essential in regulated sectors and useful everywhere else. The ICO's guidance on AI and data protection emphasises explainability, and RAG makes that dramatically easier. Fine-tuned models are opaque by comparison - you cannot point to "where" a particular answer came from in the weights.

Hallucination profiles differ too. RAG does not eliminate hallucination but it reduces it substantially when retrieval is good and the prompt forces the model to answer only from context (and refuse when context is insufficient). Fine-tuning on its own does not reduce hallucination and can make it worse if the training data is inconsistent.

Cost sits on both sides but breaks down differently. Fine-tuning has a large upfront cost (data preparation, training compute, evaluation) and low inference cost - you often end up on a cheaper base model than you would otherwise need. RAG has a lower upfront cost but higher per-query cost because every query includes retrieved context, which inflates token counts.

When to pick fine-tuning

Fine-tuning is the right call when you need consistent, structured behaviour that is hard to specify in a prompt. Classification into a large, specific taxonomy is a classic case - if you have 200 support ticket categories and inconsistent labelling from prompt engineering alone, a fine-tune will outperform prompt-based few-shot every time, at lower inference cost.

Structured extraction is another. If you need to pull 40 fields from unstructured invoices, contracts, or clinical notes with high consistency, fine-tuning on a few thousand annotated examples is usually the fastest path to production accuracy.

Tone and voice reliability is a legitimate use case, particularly for consumer-facing agents where inconsistency is jarring. A fine-tune can lock in a brand voice more reliably than a system prompt, especially over long conversations where prompt influence tends to decay.

Domain reasoning patterns - legal argument structure, medical differential diagnosis format, financial analysis frameworks - can be taught through fine-tuning where the pattern is stable and well-exemplified.

You also want fine-tuning when latency and cost per query matter enormously. A well-fine-tuned smaller model can replace GPT-4 for a specific task at a tenth of the cost and half the latency. For high-volume workloads (millions of queries per month) this pays back the training investment within weeks.

What you need for fine-tuning to work: at least a few hundred high-quality examples for simple tasks, a few thousand for anything nuanced, a held-out evaluation set that reflects production distribution, and someone willing to iterate on the data rather than the hyperparameters. Data quality dominates everything else.

When to pick RAG

RAG is the right call when the model needs to answer questions grounded in a body of knowledge that changes, is large, or requires citation. Customer-facing support assistants over a product knowledge base are the canonical example. Internal assistants over policy documents, wikis, or Confluence spaces are another. Search-and-summarise over legal filings, research papers, or case history is a third.

Any use case where being wrong has a compliance cost pushes hard towards RAG. Financial services queries against regulatory documents, healthcare queries against clinical guidelines, legal queries against case law - all need traceable sourcing, and RAG provides it natively.

RAG is also the answer when the knowledge base updates frequently. A product support bot that has to reflect this week's release notes cannot wait for a training cycle. Re-embedding new documents takes minutes.

What you need for RAG to work well: clean, well-structured source documents (garbage in, garbage out applies harder here than most people think), a chunking strategy that preserves semantic units, hybrid retrieval (dense + sparse), a reranker for anything beyond trivial corpora, and an evaluation harness measuring retrieval precision/recall separately from generation quality. Skipping the evaluation harness is the single most common way we see RAG projects fail.

When to combine them

The most sophisticated production systems use both. A fine-tuned model handles a specific behaviour - say, following a strict refusal policy, producing structured JSON output, or reasoning in a domain-specific format - while RAG supplies the current facts.

A common architecture: fine-tune a smaller open-weight model (Llama 3.1 8B, Qwen 2.5) on domain reasoning and output format, then wire it into a RAG pipeline that retrieves from the current knowledge base. You get low inference cost, consistent behaviour, current knowledge, and citations. The initial build is heavier but the unit economics at scale are compelling.

The other combination worth flagging is fine-tuning specifically to teach a model how to use retrieved context better. Base models are surprisingly variable at following instructions like "answer only from the provided context and cite the source." A small fine-tune on examples of good context-grounded answers can substantially lift RAG output quality without changing the retrieval layer at all.

Cost and effort: rough numbers

For a typical mid-market build, fine-tuning a hosted model (OpenAI, Anthropic, Google) runs £5-25k in data preparation and training costs for a first version, plus ongoing evaluation and retraining as the task evolves. Fine-tuning an open-weight model on your own infrastructure is cheaper on compute but heavier on engineering time - budget £15-50k for a production-grade pipeline including MLOps.

A production RAG system typically lands at £20-80k for a first build depending on data volume, source system complexity, and evaluation rigour. The retrieval layer is where cost concentrates - clean document ingestion, hybrid search, reranking, and eval harness. Inference costs then run monthly at whatever your query volume dictates, typically £500-5k/month for mid-market usage.

Both approaches require ongoing operation. RAG needs someone monitoring retrieval quality as the knowledge base grows. Fine-tuned models need retraining as the task drifts. Neither is fire-and-forget, and pretending otherwise is how these projects end up abandoned.

How to decide in practice

Start with the question: is the problem knowledge, behaviour, or both? If it is purely knowledge and the knowledge changes, RAG. If it is purely behaviour and the behaviour is stable, fine-tuning. If it is both, plan for a combined system but build RAG first - it is easier to reason about, cheaper to iterate, and gives you a working baseline against which to measure whether fine-tuning is actually adding value.

Prompt engineering is the free option that both techniques should be measured against. A serious prompt with well-chosen few-shot examples solves more problems than most teams expect. Only move to fine-tuning or RAG when you have a specific, measurable gap that prompting cannot close.

Instrument everything from day one. Track answer quality, retrieval hit rate, refusal accuracy, latency, cost per query, and user feedback. Without measurement, you cannot tell whether your fine-tune is working or your RAG is drifting, and you will make the wrong architecture call on the next project too.

Frequently asked questions

Can RAG replace fine-tuning entirely?

For most knowledge-retrieval use cases, yes. RAG handles the majority of "answer questions about our documents" scenarios without any fine-tuning at all, and the base models have improved to the point where a well-designed RAG system on GPT-4o or Claude Sonnet often matches what fine-tuning would produce, at lower total cost. Where RAG cannot replace fine-tuning is when you need specific output structure, tone consistency at scale, or classification behaviour that is genuinely hard to specify in a prompt. Start with RAG and only add fine-tuning when you have measured a specific gap.

How much training data do I need to fine-tune effectively?

For hosted fine-tuning of a strong base model, you can get useful results from 50-100 high-quality examples for narrow tasks, though 500-2000 is more typical for production. For open-weight models trained from scratch on a domain, you are looking at 5000+ examples minimum, and often 20000+ for nuanced work. Data quality dominates quantity by a wide margin. A thousand carefully curated examples will beat ten thousand noisy ones every time. Budget more time for data preparation than for training itself - most fine-tuning projects that fail, fail on the dataset.

Does fine-tuning teach the model new facts?

Unreliably, and you should not depend on it. Fine-tuning shifts model behaviour and can reinforce facts the base model already partly knows, but it is a poor mechanism for injecting fresh factual knowledge. The model may appear to have learned a fact during training but hallucinate confidently on adjacent questions in production. For factual knowledge, use RAG. Fine-tuning is for teaching patterns of behaviour, not for teaching the model what your product does, what your policies say, or what happened in your last quarterly report.

What are the GDPR and data protection implications?

Both approaches involve personal data risks that need proper assessment under UK GDPR. Fine-tuning bakes training data into model weights in a way that is effectively irreversible, which creates challenges for data subject rights (particularly right to erasure) and needs careful data minimisation before training. RAG keeps source documents separate from the model, which makes deletion and audit straightforward but requires proper access controls on the retrieval layer so users cannot query documents they should not see. The ICO's guidance on AI and data protection is the definitive reference, and a DPIA is almost always required before either approach goes to production with personal data.

How long does each take to build?

A first production RAG system on a well-scoped knowledge base typically takes 6-12 weeks from kickoff, including ingestion pipelines, retrieval, evaluation harness, and a usable interface. A first fine-tune of a hosted model can be done in 2-4 weeks if the training data already exists in usable form, or 6-10 weeks if data preparation is part of the project. Combined systems take 10-16 weeks. In both cases, the first version is the start of the work, not the end - budget for at least three months of iteration and evaluation after go-live before the system is genuinely mature.

Can I fine-tune on top of RAG output?

Yes, and it is an underused technique. You can generate a synthetic training dataset by running your RAG pipeline over a set of representative queries, capturing high-quality outputs (either the best RAG answers or human-edited versions), and fine-tuning a smaller model to reproduce that quality without the retrieval overhead. This works particularly well for high-volume, cost-sensitive workloads where you have a strong RAG baseline and want to compress it into a cheaper, faster model. It does not replace RAG for knowledge that changes, but it can dramatically reduce inference cost for stable domains.

Which vendors and tools should I use?

For hosted fine-tuning, OpenAI and Google are the main paths for closed models, with strong tooling around evaluation and deployment. For open-weight fine-tuning, Hugging Face TRL, Axolotl, and Unsloth are the common choices, running on either your own GPUs or a service like Modal, Replicate, or RunPod. For RAG, the retrieval stack is more fragmented - Postgres with pgvector is the pragmatic default for mid-market builds, with Pinecone, Weaviate, or Qdrant for larger scales. LangChain and LlamaIndex are useful for prototyping but most production systems end up with more direct implementations. The right choice depends on your existing stack, data residency requirements, and team skills more than on any one tool being objectively best.

Making the call

Fine-tuning and RAG are complementary tools, not competing ones. The teams that get this right pick the technique that fits the specific problem, measure ruthlessly, and iterate. The teams that get it wrong pick based on what they read on Hacker News, over-invest in one approach, and blame the model when it does not work.

If you are scoping a build and want a second opinion on whether fine-tuning, RAG, or a combined architecture fits your use case - and what it would actually cost - AI Advisory runs discovery engagements that produce a costed, buildable technical plan rather than a slide deck. Get in touch to talk through what you are trying to build.

Further reading

Sources referenced for context not directly cited in the body:

Ready to put this into production? book a discovery call.

• ICO - Explaining decisions made with artificial intelligence (Part 1: The basics of explaining AI)