Fine-Tuning vs RAG: How to Choose the Right Approach

Fine-tuning and Retrieval-Augmented Generation (RAG) are the two dominant techniques for adapting a general-purpose large language model to a specific business problem. They are often presented as competitors. They are not. They solve different problems, cost different amounts, and fail in different ways. Choosing the wrong one is the most common expensive mistake we see in AI projects that stall at proof-of-concept.

This article explains what each technique actually does, when to reach for which, how the costs compare, and how they combine. It assumes you have a working understanding of what an LLM is and are trying to decide how to make one useful for your organisation.

The one-line difference

Fine-tuning changes the model's weights so it behaves differently. RAG leaves the model alone and gives it relevant documents at query time so it can answer with information it did not previously have.

Put another way: fine-tuning teaches the model a new skill or style. RAG gives it new facts. The two are complementary, not substitutes, and the confusion between them wastes an enormous amount of budget.

If your problem is that the model does not know your product catalogue, your policies, or last week's board minutes, RAG is almost certainly the answer. If your problem is that the model does not respond in your house voice, does not follow your specific structured output format, or cannot perform a specialised task like extracting entities from clinical notes, fine-tuning is a plausible answer.

What RAG actually does

RAG works in two stages. First, at query time, a retrieval system searches a corpus of your documents (support tickets, product docs, contracts, whatever) and finds the passages most relevant to the user's question. Second, those passages are inserted into the prompt sent to the LLM, along with an instruction like "answer the question using only the context provided". The model then generates a response grounded in that retrieved context.

The retrieval step usually combines dense vector search (embeddings stored in a vector database like pgvector, Pinecone, or Weaviate) with keyword search (BM25). Hybrid retrieval consistently outperforms either alone in production evaluations - Microsoft's Azure AI Search team has published benchmarks showing hybrid + reranking lifting retrieval quality by 20-40% over dense-only on realistic enterprise corpora.

What RAG gives you:

• Fresh knowledge. Update the source documents, and the model's answers update immediately. No retraining cycle.
• Citations. You can show the user which document each answer came from, which is essential in regulated sectors and useful everywhere else.
• Access control. You can filter retrieval by the user's permissions, so the model only sees documents that user is allowed to read.
• Low upfront cost. A competent team can build a working RAG prototype in 2-4 weeks.

What RAG does not give you:

• Behavioural change. The model still writes the way the base model writes.
• Compression. Every query pays token costs for the retrieved context.
• Novel task performance. If the base model cannot do the underlying task (say, classify a medical image or reason about a proprietary programming language), retrieving more text does not fix that.

What fine-tuning actually does

Fine-tuning takes a pre-trained base model and trains it further on your data, adjusting its weights. There are three main flavours in current practice:

Supervised fine-tuning (SFT) trains the model on input-output pairs. You show it 500-50,000 examples of "here is the input, here is the ideal output", and the model learns to produce outputs that look like your examples. This is the workhorse for style, format, and task-specific tuning.

Preference tuning (DPO, RLHF) trains the model on pairs of "better" and "worse" responses. This is how frontier labs align models to be helpful and harmless, and it is occasionally worth doing for domain-specific quality bars, but it is expensive and data-hungry.

Parameter-efficient fine-tuning (LoRA, QLoRA) trains a small set of adapter weights instead of the full model. This is dramatically cheaper - a LoRA on a 7B parameter open model can be trained on a single consumer GPU in a few hours - and is now the default for most practical fine-tuning outside frontier labs.

What fine-tuning gives you:

• Consistent style and format. If you need every output to be valid JSON matching a specific schema, or every email to sound like your brand voice, fine-tuning outperforms prompting for consistency at scale.
• Lower per-query cost. A fine-tuned smaller model can often match a prompt-heavy larger model on a narrow task, at 10-20% of the inference cost.
• Learned tasks. Genuinely novel behaviours that cannot be reliably prompted - specialised classification, structured extraction from noisy inputs, translation into a low-resource dialect.

What fine-tuning does not give you:

• New facts. Fine-tuning is a poor way to teach a model "our returns policy changed on 1 March". The knowledge gets buried in the weights, is hard to update, and the model will happily contradict it under the right prompt.
• Auditability. You cannot point at a fine-tuned answer and say which training example produced it.
• Cheap iteration. Each change means re-running training, re-evaluating, and re-deploying.

Cost comparison

Rough order-of-magnitude figures for a mid-market build, based on our recent projects and public vendor pricing:

RAG. Build cost £25k-£80k for a first production system covering a defined corpus. Ongoing costs are dominated by inference (£0.50-£5 per 1,000 queries depending on model and context size) plus vector database hosting (£100-£1,500/month depending on scale). Content updates are near-free - you just re-index changed documents.

Fine-tuning via API (OpenAI, Anthropic, Google). Training cost is modest, typically a few hundred to a few thousand pounds for a supervised fine-tune of a mid-size model on 5,000-20,000 examples. Inference on the fine-tuned model is generally 1.5-6x the base model rate per token. Full cost is dominated by the human effort of building and cleaning the training set, which is where most projects underestimate: budget £15k-£50k for a serious training-data effort.

Fine-tuning via open-source (Llama, Mistral, Qwen with LoRA). Training is cheap - a few hundred pounds of GPU time. Hosting the resulting model is the cost driver: £500-£5,000/month for a production-grade deployment on Modal, Together, Fireworks, or your own infrastructure. This route makes sense when you have real volume (hundreds of thousands of queries/month) and when data residency or IP concerns rule out API fine-tuning.

Where teams get burned: they attempt fine-tuning to solve a knowledge problem, discover it does not work, then rebuild as RAG. That is a £40k-£150k detour we see roughly quarterly.

A decision framework

Work through these questions in order:

1. Does the problem involve knowledge that changes or is proprietary? If yes, RAG. Product docs, policies, tickets, contracts, meeting notes, CRM data - all RAG territory. Do not fine-tune on this.

2. Is the base model already competent at the underlying task, and you mainly need it to know your specific stuff? RAG. This covers most customer support, internal Q&A, research assistants, and document search use cases.

3. Do you need consistent structured output that prompting cannot reliably produce? Consider fine-tuning. Also consider structured output APIs (OpenAI's structured outputs, Anthropic's tool use, Instructor library) before committing to a fine-tune - they solve 80% of format-consistency problems without training.

4. Do you need a specialised behaviour or style at high volume? Fine-tuning becomes attractive. Classifying millions of support tickets by intent, extracting entities from clinical notes, writing marketing copy in a very specific voice - these justify the effort.

5. Is inference cost a bottleneck? A fine-tuned smaller model can dramatically reduce per-query cost. But only pursue this once you have a working system with real usage data - premature optimisation here is common and wasteful.

6. Do you need both? Often, yes. A fine-tuned model that has learned your style and structure, combined with RAG that supplies the current facts, is a common production pattern - especially for customer support and internal assistants at scale.

The hybrid pattern most production systems end up using

In practice, the mature production architecture for a serious LLM application usually looks like this:

1. Base model choice. Start with a strong general model (GPT-4o, Claude Sonnet, Gemini Pro, or Llama 3.1 70B for self-hosted).
2. RAG layer. Hybrid retrieval over your corpus, with a reranker (Cohere Rerank, or a fine-tuned cross-encoder) and passage filtering.
3. Prompt engineering. Careful system prompts, few-shot examples, structured output schemas.
4. Evaluation harness. A held-out test set of 200-1,000 realistic queries, scored by both automated metrics and human review, run on every change.
5. Optional fine-tune. Only after the above is working, if evaluation shows a persistent gap that prompting and retrieval cannot close.

The order matters. Roughly 80% of the systems we build reach production quality without ever fine-tuning the LLM. The ones that do fine-tune benefit precisely because the RAG and evaluation infrastructure was already in place to tell them what to fine-tune for.

Common failure modes

Fine-tuning to teach facts. The single most common mistake. Someone reads that fine-tuning "customises the model" and assumes that means teaching it about their business. It does not, not reliably. The model will regurgitate training examples verbatim, hallucinate confidently about details, and go stale the moment your data changes.

RAG without evaluation. Building a RAG pipeline is straightforward. Building one that gives correct, grounded, non-hallucinated answers on your specific corpus is not. Without an evaluation harness, teams ship RAG systems that look impressive in demos and produce embarrassing errors in production. Budget at least 20% of build effort for evaluation.

Fine-tuning on synthetic data generated by the model you are fine-tuning. Tempting, and occasionally useful, but the pathologies are well-documented - the model amplifies its own biases and degrades on the long tail. Human-curated or human-verified data is worth the cost.

Ignoring the base model upgrade cycle. A fine-tune on GPT-3.5 in 2023 is now inferior to base GPT-4o-mini on almost every task, and the fine-tune has to be redone. Frontier base models improve faster than most fine-tuning gains hold their value. Assume any fine-tune has a 6-18 month useful life.

GDPR and compliance considerations

For UK organisations, the choice between RAG and fine-tuning has real implications under UK GDPR and the ICO's guidance on AI and data protection.

RAG keeps your data in your database. The LLM sees only the retrieved snippets at query time. If you use a UK/EU-hosted model or an API with appropriate contractual and residency arrangements (Azure OpenAI in UK South, AWS Bedrock in eu-west-2, or a self-hosted open model), you can maintain a defensible data-processing story. Personal data can be redacted before retrieval, access-controlled per user, and deleted from the corpus when a data subject requests erasure.

Fine-tuning is harder. Training data becomes embedded in model weights in a way that is not reversibly extractable. The ICO's guidance on AI and data protection is clear that data minimisation and the right to erasure still apply. Fine-tuning on personal data creates a compliance problem that RAG does not. For most enterprise use cases, this alone is a strong argument for RAG-first architectures.

Frequently asked questions

Can I use both RAG and fine-tuning together?

Yes, and this is often the strongest architecture for production systems at scale. Fine-tune the model to learn your specific task, format, or style - for example, how to structure a support response, or how to extract fields from a specific document type. Then use RAG at query time to give it the current, specific facts it needs. The fine-tuning handles the "how", the RAG handles the "what". Most large-scale production LLM applications converge on this pattern once they mature past the initial build phase.

How much data do I need to fine-tune a model?

For supervised fine-tuning on a specific task, 500-1,000 high-quality examples is a useful minimum, 5,000-20,000 is a comfortable range, and beyond about 50,000 you hit diminishing returns for most tasks. Quality matters vastly more than quantity - 1,000 carefully curated examples will outperform 20,000 noisy ones. LoRA fine-tuning can work with even fewer examples for narrow style transfer. Before assembling training data, invest in your evaluation set first: 200-500 realistic held-out examples that reflect production traffic, so you can measure whether the fine-tune actually helps.

Is RAG going to be replaced by long-context models?

No, though the boundary shifts. Frontier models now support 1M+ token contexts, so "just stuff everything in the prompt" is technically possible for small corpora. In practice, RAG remains essential for three reasons: cost (paying for 500,000 tokens per query is not viable at scale), latency (long contexts are slow), and accuracy (models get worse at finding specific facts as context grows - the "lost in the middle" problem is well-documented). For corpora under a few hundred pages that fit comfortably in context, long-context can replace RAG. For anything larger or higher-volume, RAG remains the pragmatic answer.

Which LLM should I fine-tune?

For UK mid-market builds where data can go to a major cloud provider, OpenAI's fine-tuning API on GPT-4o-mini is the pragmatic default - good base capability, reasonable cost, mature tooling. If you need self-hosting for data residency or cost reasons, Llama 3.1 8B or 70B with LoRA via Unsloth or Axolotl is the current sensible choice, hosted on Modal, Together, Fireworks, or your own GPUs. Mistral and Qwen are also credible. Avoid fine-tuning Claude - Anthropic does not currently offer public fine-tuning for most customers. Whatever you pick, assume you will need to re-fine-tune on a newer base model within 12-18 months.

What does a RAG build actually cost end-to-end?

For a first production RAG system covering a defined corpus with authentication, evaluation, and monitoring, budget £30k-£80k for build and 8-14 weeks of elapsed time. That covers document ingestion pipelines, hybrid retrieval, reranking, LLM integration, an evaluation harness, a basic admin UI, and deployment. Ongoing costs run £500-£3,000/month for infrastructure and LLM inference for typical mid-market usage, plus ongoing content curation. Adding new document sources or use cases post-launch is usually £5k-£20k per addition. Cheaper builds exist but tend to skip evaluation infrastructure, which is where they fail in production.

Can I do this in-house or do I need an agency?

If you have an engineering team that includes at least one person with hands-on LLM production experience (not just prototypes), in-house is viable for RAG. The tooling is mature. Fine-tuning in-house is harder and depends on whether you have anyone who has trained a model before - the gap between "ran the training script" and "shipped a fine-tune that improved production metrics" is wide. Where agencies genuinely add value is the evaluation harness, the retrieval quality tuning, and the pattern library from having built the same thing 20 times. Many mid-market organisations use an agency for the first build and take it in-house for operations and iteration.

How do I know if my RAG system is actually working?

Build the evaluation set before you build the pipeline. Collect 200-500 realistic questions with expert-written correct answers, ideally drawn from real user queries or anticipated ones. Score each generated answer on retrieval quality (did it find the right passages?), faithfulness (does the answer stick to what the retrieved passages say?), and answer quality (would an expert accept it?). Automated scoring with an LLM judge is useful for regression testing between builds but should be calibrated against human review. If you cannot answer "is this system working?" with a number, you do not have a system - you have a demo.

When is fine-tuning definitely the wrong answer?

When the knowledge is changing (documents update, policies change, prices move), when auditability matters (regulated sectors, high-stakes decisions), when you need access control on who can see what, when your training data would include personal data you may need to delete under GDPR, when volume is low enough that inference cost is not a real constraint, and when you have not yet built a working prompted or RAG baseline to measure improvement against. Any one of these is a strong signal to solve the problem another way. Two or more, and fine-tuning is almost certainly a mistake.

Making the right choice

The honest short version: for most UK mid-market AI projects, RAG plus strong prompting plus a rigorous evaluation harness gets you to production. Fine-tuning is a specialist tool for specific problems - style consistency at scale, structured extraction, cost optimisation on high-volume narrow tasks - and it is worth its cost when those problems are real. It is not a general-purpose way to "teach the model about our business", and treating it as one is the fastest way to burn a six-figure budget with nothing shipped.

If you are trying to decide between RAG and fine-tuning for a specific project, or you have a stalled AI proof-of-concept that needs a second opinion, AI Advisory runs a two-week strategy engagement that ends with a costed, buildable recommendation. Get in touch to scope it.

Further reading

Sources referenced for context not directly cited in the body:

Ready to put this into production? book a discovery call.

• RAG vs. Fine-tuning - IBM
• RAG vs. Fine-Tuning: How to Choose - Oracle