RAG Analysis: What It Is, How It Works, and When to Use It

RAG analysis is the practice of examining how a Retrieval-Augmented Generation system finds, ranks, and uses information to answer a question. It sits at the intersection of search engineering and applied LLM work: you are asking not just "did the model give a good answer?" but "did it retrieve the right evidence, did it use that evidence faithfully, and can I prove it?"

If you are researching this term, you are probably in one of three positions. You are evaluating a RAG system a vendor built for you. You are building one and need to know how to measure it. Or you are trying to work out whether RAG is the right pattern at all versus fine-tuning, prompt engineering, or a plain search index. This article covers all three.

What RAG actually is

Retrieval-Augmented Generation, introduced by Lewis et al. at Facebook AI Research in 2020, is an architecture that combines two components. First, a retrieval system fetches relevant documents from a knowledge source, typically using vector similarity search over embeddings, keyword search (BM25), or a hybrid of both. Second, a generative language model takes the retrieved documents plus the user's question and produces an answer grounded in that retrieved context.

The point of RAG is to let a general-purpose LLM answer questions about specific, private, or recent information it was never trained on. A GPT-4 or Claude model does not know your contract terms, your product catalogue, or last week's board minutes. RAG bolts a searchable index of that content onto the model at query time, so the answer reflects your data rather than the model's training corpus.

RAG analysis, then, is the discipline of measuring whether each stage is doing its job. Retrieval quality, generation faithfulness, and end-to-end answer quality are three distinct things, and confusing them is the most common reason RAG projects stall.

The anatomy of a RAG pipeline

Before you can analyse a RAG system, you need a shared mental model of what happens between question and answer. A production pipeline typically has these stages:

Ingestion and chunking

Source documents (PDFs, web pages, database records, transcripts) are broken into chunks, usually 200-800 tokens each, with some overlap. Chunk boundaries matter more than most teams realise: split a table across two chunks and neither will make sense in isolation.

Embedding

Each chunk is converted to a vector using an embedding model such as OpenAI's text-embedding-3-large, Cohere Embed v3, or an open-source model like BGE or E5. These vectors are stored in a vector database (Pinecone, Weaviate, Qdrant, or Postgres with pgvector).

Retrieval

At query time, the user's question is embedded using the same model, and the vector store returns the top-k most similar chunks. Serious systems combine this with BM25 keyword search and a reranker (like Cohere Rerank or a cross-encoder) that reorders the shortlist.

Generation

The retrieved chunks are placed into a prompt template alongside the user's question and system instructions, then sent to the LLM. The model produces an answer, ideally with citations back to specific chunks.

Post-processing

Guard rails check for refusal conditions, hallucination markers, or policy violations. Citations are rendered. The answer is returned to the user.

Every one of these stages can go wrong independently. RAG analysis means being able to point at any given failure and say which stage caused it.

How to analyse retrieval quality

Retrieval is the foundation. If the right chunks never make it into the context window, the LLM cannot produce a correct answer, no matter how capable it is. Most teams that complain their RAG system "hallucinates" are actually looking at a retrieval failure the generation stage tried to cover for.

The two core retrieval metrics are borrowed directly from information retrieval research:

Recall@k measures the proportion of relevant documents that appear in the top k retrieved results. If your gold set for a question contains three relevant chunks and two show up in the top 10, recall@10 is 0.67. Low recall means the right information exists in your index but is not being surfaced.

Precision@k measures the proportion of retrieved documents that are relevant. Precision matters because context windows are finite and irrelevant chunks crowd out useful ones, and because they can actively mislead the generator.

For ranked results, teams also use Mean Reciprocal Rank (MRR), which rewards systems for putting the first relevant result high, and NDCG, which handles graded relevance judgements. The RAGAS evaluation framework, published in 2023, packaged several of these metrics specifically for RAG pipelines and is worth reading before you build your own harness.

To run these evaluations you need a labelled dataset: questions paired with the chunks that should be retrieved to answer them. Building this manually is tedious but essential. Fifty well-labelled questions are more useful than five thousand synthetic ones.

How to analyse generation quality

Once retrieval is doing its job, the question is whether the LLM uses the retrieved context correctly. Three properties matter:

Faithfulness (or groundedness) is whether every claim in the answer is supported by the retrieved context. A faithful answer may be incomplete, but it does not invent facts. The standard method is to decompose the answer into atomic claims and, for each one, ask a judge model whether the claim is entailed by the retrieved chunks.

Answer relevance is whether the answer addresses the question that was asked. A system can produce a beautifully grounded answer to the wrong question, which is common when the retriever pulls tangentially related content.

ND context utilisation asks whether the model used the best chunks it was given. Sometimes the right evidence is in position 3 of the context but the model anchored on position 1. This is where reranking and prompt design earn their keep.

These are typically measured using LLM-as-judge evaluation: a stronger model (GPT-4 class) scores each answer against reference criteria. Anthropic and OpenAI both publish guidance on structuring these evaluations to reduce judge bias. Human review of a sample is still non-negotiable for anything customer-facing.

End-to-end evaluation and what "good" looks like

Retrieval and generation metrics tell you where problems are. End-to-end evaluation tells you whether the system meets the user need. This is where most vendor demos gloss over the hard part.

A robust end-to-end harness contains:

• A frozen evaluation set of 100-500 questions covering the real distribution of user queries, including edge cases, adversarial phrasings, and out-of-scope questions the system should refuse.
• Reference answers written by subject-matter experts for at least a subset, with acceptable variations documented.
• Automated metrics (faithfulness, relevance, retrieval recall) run on every deployment.
• Human review of a rolling sample, ideally by the people who will use the system.
• Regression tracking: you must be able to say whether last week's change made the system better or worse, and on which query types.

Benchmarks published by Anthropic, OpenAI, and the RAGAS team suggest well-tuned RAG systems on well-defined domains achieve faithfulness scores of 0.85-0.95 and answer relevance of 0.80-0.90. Below 0.75 on either metric, you have a system that is not ready for external users. Above 0.95 sustained across a diverse eval set is rare and usually indicates the eval set is too easy.

One warning: metrics move together in unexpected ways. Improving retrieval recall by increasing k often reduces precision and can hurt faithfulness because the model gets confused by conflicting chunks. This is why RAG analysis needs to look at the whole pipeline, not just tune one number.

Common failure modes and how to diagnose them

Certain failure patterns show up in almost every RAG deployment. Recognising them shortens debugging time from weeks to hours.

Chunking artefacts. The answer is technically in the corpus but split across two chunks in a way that destroys meaning. Fix: revisit chunk size, add hierarchical or semantic chunking, include document titles and section headers in every chunk.

Embedding drift. The user asks about "holiday allowance" but the corpus says "annual leave entitlement." Off-the-shelf embeddings sometimes miss this. Fix: hybrid search with BM25, domain-tuned embeddings, or query expansion.

Lost in the middle. LLMs pay less attention to content in the middle of long contexts, a phenomenon Liu et al. documented in 2023. Fix: rerank so the strongest chunks are first and last, keep contexts tight.

Refusal failure. The system answers questions it should decline (out of scope, insufficient evidence). Fix: explicit refusal instructions in the system prompt, a retrieval-confidence threshold, and a separate classifier for out-of-scope queries.

Stale index. Documents changed but the index did not. Fix: incremental re-indexing pipelines and a metric that tracks index freshness.

Prompt regression. Someone tweaked the system prompt and quality dropped on question types no one tested. Fix: never ship prompt changes without running the full eval harness.

When RAG is the right pattern, and when it is not

RAG is not the answer to every LLM problem. It solves a specific set of problems well and is a poor fit for others.

Use RAG when the answer depends on a body of text that is too large to fit in context, changes often, or is private to your organisation. Customer support over a product knowledge base, internal search over policies and contracts, and research assistants over technical literature are natural fits.

Do not use RAG when the task is about reasoning or style rather than recall. If you need the model to write in a particular tone, follow a complex procedure, or apply judgement, prompt engineering and few-shot examples typically get you further. Fine-tuning is the right tool when you have thousands of examples of the input-output behaviour you want and the underlying model cannot be coached to produce it consistently.

Many production systems combine all three: fine-tuning for behaviour, prompt engineering for structure, RAG for knowledge. The ICO's guidance on AI and data protection is worth reading before you push private data through any of these architectures, particularly around lawful basis for processing and the transparency obligations under UK GDPR.

Frequently asked questions

How is RAG analysis different from evaluating a normal LLM output?

Evaluating a standalone LLM output focuses on whether the response is correct, coherent, and appropriate. RAG analysis adds two upstream questions: was the right evidence retrieved, and did the model faithfully use that evidence? A RAG system can produce a fluent, plausible answer that is completely disconnected from the retrieved context, or retrieve perfect evidence and then ignore it. You cannot diagnose either failure without instrumenting retrieval and generation separately. This is why RAG evaluation frameworks like RAGAS report three metrics minimum (faithfulness, answer relevance, context recall) rather than a single quality score.

What tools do teams use to run RAG analysis in practice?

The most common open-source frameworks are RAGAS, TruLens, and DeepEval, all of which package the standard metrics and integrate with LangChain and LlamaIndex. For observability in production, teams use LangSmith, Langfuse, Arize Phoenix, or Weights & Biases Weave to capture traces and run evaluations on live traffic. For labelled dataset creation, Argilla and Prodigy are popular. Serious teams eventually build a thin custom layer on top of these because their evaluation criteria are domain-specific and vendor frameworks assume general use cases. Budget one to two engineering weeks for a first-pass evaluation harness, more if your domain requires expert review.

How large does my evaluation set need to be?

Start with 50 questions covering the main query types you expect, hand-labelled with reference answers or expected retrieval chunks. This is enough to catch major regressions and drive early iteration. Grow to 200-500 as the system approaches production, adding adversarial cases, out-of-scope questions, and edge cases surfaced from real user logs. Beyond 500, returns diminish unless you are covering genuinely different scenarios. The quality and diversity of your evaluation set matters far more than its size: 100 well-chosen questions from real users beat 5,000 synthetic questions generated by GPT-4 every time.

Can I use an LLM to evaluate my RAG system, or do I need humans?

Both, in different proportions at different stages. LLM-as-judge evaluation (using GPT-4 or Claude to score answers) is fast, cheap, and correlates reasonably well with human judgement for faithfulness and relevance, provided you structure the prompts carefully and use a stronger model than the one being evaluated. It is essential for the volume of evaluation needed during development. Human review remains necessary for calibrating the judge, for final sign-off before launch, and for high-stakes domains like healthcare or legal. A common split is 100% automated evaluation on every commit, 10% human sample review weekly, and full human review at major milestones.

How much does building a proper RAG evaluation harness cost?

For a mid-market build, expect £8,000-£25,000 of engineering time to get a working harness in place, plus ongoing SME time to create and maintain the evaluation set. That covers labelled dataset creation, integration of an evaluation framework, dashboards for retrieval and generation metrics, and CI integration so evaluations run on every deployment. It is one of the highest-return investments in a RAG project because it turns "the system feels better" into a measurable claim, and it makes vendor and model swaps rational rather than political. Skipping it is the single most common reason RAG projects fail to move past pilot.

Does RAG solve hallucination?

It reduces hallucination significantly but does not eliminate it. When retrieval works and the prompt is well-constructed, faithfulness scores of 0.90+ are achievable, meaning roughly 9 in 10 answers are fully grounded in retrieved evidence. The remaining failures come from the model paraphrasing incorrectly, combining information from multiple chunks in invalid ways, or filling gaps with prior knowledge when the retrieved context is incomplete. Mitigations include stricter prompts that require citation for every claim, refusal patterns when confidence is low, and post-generation verification steps. RAG is a substantial improvement over ungrounded generation, not a guarantee of factual accuracy.

How does RAG interact with GDPR and UK data protection law?

Any RAG system processing personal data falls under UK GDPR, and the ICO expects the same standards of lawful basis, transparency, and data minimisation as any other processing activity. Practical implications: document what personal data goes into your index, ensure you have a lawful basis for embedding and storing it, respect deletion requests (which means being able to remove documents from the vector store, not just the source), and consider whether sending retrieved context to a third-party LLM API constitutes an international transfer requiring safeguards. The ICO's guidance on AI and data protection, updated in 2024, is the authoritative reference. For sensitive domains, self-hosted models and vector stores are increasingly viable.

How long does it take to move a RAG proof of concept to production?

A working demo can be built in a few days using LangChain or LlamaIndex against a small corpus. Getting to production for a mid-market use case typically takes 8-16 weeks, with the bulk of that time spent on evaluation, chunking strategy, retrieval tuning, refusal patterns, and integration with existing systems rather than on the LLM itself. Teams that skip evaluation and ship the demo tend to be back rebuilding within six months. The pattern that works is: two weeks to a demo, four weeks to instrumented evaluation, four to eight weeks tuning against real usage, then production release with observability in place.

Where to go from here

RAG analysis is the difference between a demo that impresses in a boardroom and a system that holds up under real user traffic. The mechanics are not exotic: chunk sensibly, retrieve with hybrid search and reranking, generate with tight prompts and citation requirements, and measure every stage against a labelled evaluation set. What separates successful RAG deployments from failed ones is discipline about that last step.

If you are scoping a RAG build, or trying to work out why an existing one is not performing, AI Advisory runs technical audits that produce a costed remediation plan against the metrics described above. Get in touch to discuss what your system needs.

Further reading

Sources referenced for context not directly cited in the body:

Ready to put this into production? book a discovery call.

• Evaluators