RAG with LangChain: How Retrieval-Augmented Generation Actually Works

Retrieval-Augmented Generation (RAG) is the pattern behind most useful enterprise AI applications shipped in the last two years. It is how you get a large language model to answer questions about your company's contracts, your product documentation, or last quarter's board pack without fine-tuning a model or hoping the base model has memorised something close enough. LangChain is the most widely adopted open-source framework for wiring RAG systems together, and it remains a sensible default even as the ecosystem has fragmented into LlamaIndex, Haystack, and a growing set of vendor-specific SDKs.

This article explains what RAG is, how LangChain implements it, and the design decisions that separate a demo from a production system. It assumes you have written some Python and know roughly what an embedding is. If you have already built a naive RAG pipeline and are wondering why it hallucinates on 30% of queries, skip to the sections on chunking, hybrid retrieval, and evaluation.

What RAG is, and why it exists

A language model generates text by predicting the next token given a context window. Everything it "knows" is either baked into its weights during training or supplied at inference time in the prompt. The weights are frozen after training, expensive to update, and opaque - you cannot point at a specific fact and ask the model where it came from. The context window is fluid, cheap to change, and auditable.

RAG is the pattern of putting the right facts into the context window at query time, so the model generates a grounded answer instead of a plausible-sounding guess. The three-step loop is: retrieve relevant documents from a knowledge store, augment the prompt with those documents, then generate the response. The retrieval step is what distinguishes RAG from ordinary prompting - it is a search problem sitting in front of a generation problem.

The pattern was formalised in the 2020 Lewis et al. paper from Facebook AI Research, but it became the default enterprise architecture in 2023 when it became clear that fine-tuning was expensive, forgetful, and rarely the right tool for keeping a model current on changing business data. RAG lets you update the knowledge base by re-indexing a document, not retraining a model.

Why LangChain, and what it actually gives you

LangChain is a Python and JavaScript framework that provides abstractions over the components of an LLM application: models, prompts, retrievers, vector stores, document loaders, output parsers, and the orchestration logic that stitches them together. For RAG specifically, it gives you two things worth having.

First, it gives you a common interface across roughly 100 vector databases and 50 LLM providers. You can prototype against a local Chroma instance and OpenAI, then swap in pgvector and Anthropic Claude with a two-line change. This portability matters when procurement inevitably vetoes your first choice of vector store or your CFO renegotiates the model contract.

Second, it gives you the LangChain Expression Language (LCEL) and LangGraph for composing chains. A RAG pipeline in LCEL is a declarative pipe of operations - retrieve, format, prompt, generate, parse - that you can inspect, stream, and instrument without writing glue code. LangGraph, added in 2024, extends this to stateful multi-step agents with proper cycle handling.

What LangChain does not do is make your retrieval good. It provides the plumbing; the quality of your pipeline depends on decisions you make about chunking, embeddings, retrieval strategy, and evaluation. The rest of this article covers those decisions.

The minimal LangChain RAG pipeline

A working RAG pipeline in LangChain has five components. Understanding what each does, and where each fails, is the whole game.

Document loader. Reads source content from PDFs, web pages, Notion, Confluence, S3, Google Drive, or a database. LangChain ships loaders for most common sources. The loader's job is to produce a list of Document objects, each with page content and metadata (source URL, page number, author, last modified).

Text splitter. Breaks documents into chunks small enough to fit in an embedding model's context and to retrieve at useful granularity. RecursiveCharacterTextSplitter is the default. Chunk size is the single most consequential parameter in the pipeline, and we come back to it below.

Embedding model. Converts each chunk to a dense vector - typically 384, 768, 1024, or 1536 dimensions - that captures semantic meaning. OpenAI's text-embedding-3-small is a strong default for English. For multilingual or specialised domains, look at Cohere's embed-v3 or open-source models like BAAI/bge-large-en-v1.5 hosted on your own infrastructure.

Vector store. Indexes the vectors for fast nearest-neighbour search. Chroma and FAISS are fine for prototypes. For production, we default to pgvector on Postgres because most clients already run Postgres, backup and access control are solved problems, and performance is competitive up to roughly 10 million vectors. Pinecone, Weaviate, and Qdrant are reasonable managed alternatives.

Retriever and generator. At query time, the user's question is embedded with the same model, the vector store returns the top-k most similar chunks, and those chunks are formatted into a prompt template alongside the question. The LLM generates an answer conditioned on the retrieved context.

In LCEL, this whole pipeline is roughly ten lines of code. The problem is that ten-line pipelines answer roughly 60-70% of questions correctly on a real corpus, and the remaining 30-40% is where the work is.

Where naive RAG breaks, and how to fix it

The failure modes are predictable. If your first prototype is disappointing, the diagnosis is almost always one of the following.

Chunks are the wrong size. Chunks that are too small (200 tokens) lose the context needed to make sense of a passage. Chunks that are too large (2000 tokens) dilute the semantic signal in the embedding and cost more at inference. The right size is domain-dependent but 400-800 tokens with 10-20% overlap is a sensible starting point for prose. For code or structured data, chunk on natural boundaries (functions, sections, table rows) rather than character counts.

Embedding-only retrieval misses keyword matches. Dense embeddings capture semantic similarity but frequently miss exact-match queries - product SKUs, error codes, legal clause numbers, proper nouns. The fix is hybrid retrieval: run BM25 (a classical lexical search) alongside vector search and combine the results with Reciprocal Rank Fusion. LangChain's EnsembleRetriever does this in about five lines. In our builds, hybrid retrieval typically improves top-5 recall by 15-25 percentage points over dense-only.

Retrieved chunks lack context. A chunk pulled from page 47 of a policy document may reference "the previous section" or use pronouns whose antecedents are elsewhere. Two fixes: parent-document retrieval (embed small chunks, return larger parent windows) and contextual retrieval, the approach Anthropic published in September 2024 where each chunk is prepended with an LLM-generated summary of its place in the document. Anthropic reported a 49% reduction in retrieval failures using contextual embeddings with reranking.

No reranking. Top-k retrieval by cosine similarity returns candidates, not answers. A cross-encoder reranker (Cohere Rerank, BGE reranker, or Voyage rerank-2) scores each candidate against the query with a heavier model and reorders them. Retrieving top-20 with vector search and reranking down to top-5 is standard practice in production RAG.

The prompt does not enforce grounding. If your prompt says "Answer the question using the context below" the model will happily fill gaps from its parametric memory. Prompts should instruct the model to answer only from the provided context, cite the source of each claim, and explicitly refuse if the answer is not present. Refusal patterns matter more than most teams realise - a system that says "I don't know" 10% of the time is more trustworthy than one that confabulates 3% of the time.

Evaluation: the part everyone skips

Most RAG projects fail not because the pipeline is wrong but because nobody defined what "working" means. You cannot optimise a system you cannot measure, and eyeballing 20 answers is not measurement.

The minimum viable evaluation is a golden set of 50-200 question-answer pairs with citations, built by a domain expert. For each question you measure two things: retrieval quality (did the correct source appear in the top-k?) and answer quality (does the generated answer match the ground truth?). Retrieval quality is measured with precision, recall, and mean reciprocal rank. Answer quality can be scored by a stronger LLM acting as judge, cross-checked against human ratings on a sample.

The Ragas framework automates most of this and integrates directly with LangChain. It scores faithfulness (does the answer only use the retrieved context?), answer relevancy, context precision, and context recall. LangSmith, LangChain's own tracing product, adds dataset management and regression testing so you can catch quality drops when someone changes the chunk size at 5pm on a Friday.

Build the evaluation set before you build the pipeline. It forces you to define scope, surfaces edge cases early, and gives you a fixed target to iterate against.

Production considerations

Moving from a working prototype to a system you can put in front of customers or regulated internal users introduces a different set of problems.

Ingestion pipelines. The prototype loads documents once. Production has to handle incremental updates, deletions, versioning, and access control. If a document is deleted from SharePoint, the corresponding vectors need to disappear from the index within an agreed SLA. Metadata filtering (only retrieve chunks the user has permission to see) is not optional in enterprise deployments.

Latency and cost. A dense retrieval, rerank, and generation cycle can easily take 3-8 seconds end-to-end. Streaming the generation helps perceived latency. Caching frequent queries, using smaller models for classification steps, and running embeddings in batches all help real latency. Cost is dominated by generation tokens; retrieval and embedding are usually rounding errors.

Observability. You need traces of every query - the retrieved chunks, the final prompt, the model response, latency at each step. LangSmith does this for LangChain applications; OpenTelemetry with a compatible backend works if you prefer vendor neutrality. Without traces you cannot debug the inevitable "why did it say that" ticket.

Compliance and data residency. For UK clients handling personal or regulated data, the choice of embedding model and LLM provider is a GDPR question, not just a performance question. Azure OpenAI in a UK region, AWS Bedrock in London, and self-hosted models on your own infrastructure are the three common answers. The ICO's guidance on AI and data protection is the reference document.

Fallbacks and refusals. Production systems need behaviour for when retrieval returns nothing relevant, when the LLM API is down, and when the user asks something out of scope. These paths are usually 20% of the code and 80% of the difference between a good and a bad system.

When RAG is the wrong answer

RAG is not always the right pattern. If your data fits in the context window, just include it - long-context models like Gemini 1.5 Pro (2 million tokens) and Claude 3.5 Sonnet (200k tokens) make this viable for many document sets. If the task is transformation rather than retrieval (summarisation, translation, extraction from a known document), you do not need retrieval at all. If you need the model to internalise a style or a reasoning pattern rather than recall facts, fine-tuning is the right tool.

The clearest signal that you need RAG is: the knowledge changes frequently, the corpus is too large for the context window, and the user's query is genuinely open-ended over that corpus. Customer support over a product catalogue, internal Q&A over policies and playbooks, legal research over case law, and technical assistants over API documentation are all textbook cases.

Frequently asked questions

Do I need LangChain to build RAG, or can I write it from scratch?

You can write RAG in about 100 lines of Python using the OpenAI SDK and a vector store client directly, and for a single-purpose system with one data source and one model that is often the right choice. LangChain earns its complexity when you need to swap components, compose multiple retrievers, add tools and agents, or instrument the pipeline for evaluation. If you expect the system to grow beyond a single use case, or if you want your team to work in a common vocabulary, LangChain or LlamaIndex is worth the abstraction cost. For throwaway prototypes, raw SDK calls are faster.

How does LangChain compare to LlamaIndex for RAG?

LlamaIndex is more opinionated about retrieval and indexing, and its data connectors and advanced retrieval strategies (recursive retrieval, sub-question decomposition) are more polished out of the box. LangChain is broader - it covers agents, tool use, and orchestration well beyond retrieval - and has stronger observability through LangSmith. In practice, many production systems use LlamaIndex for the retrieval layer and LangChain for orchestration and agent logic. They are not mutually exclusive. Pick LlamaIndex if RAG is the whole product; pick LangChain if RAG is one component of a larger agentic system.

What does a typical RAG build cost and how long does it take?

For a first production RAG system on a defined corpus (say, 10,000-100,000 documents) with a web or Slack front end, expect 8-14 weeks and £40k-£120k depending on data quality, integration complexity, and evaluation rigour. The build itself is rarely the expensive part - the majority of effort goes into ingestion (getting data out of messy source systems), evaluation set construction, and iterating on retrieval quality against real user queries. Ongoing running costs are typically £500-£5,000 per month in model and infrastructure fees for mid-market usage volumes, before support and iteration retainers.

Which vector database should I use?

For most UK mid-market builds we default to pgvector on managed Postgres. It performs well up to roughly 10 million vectors, uses infrastructure the client already operates, and inherits their existing backup, monitoring, and access-control setup. Pinecone is the easy managed choice if you want zero operational burden and can accept US or EU hosting. Qdrant and Weaviate are strong open-source options if you need advanced filtering or hybrid search built in. Chroma and FAISS are fine for local development but not production. The vector store is rarely the bottleneck - retrieval strategy matters more than the specific database.

How do we stop the model from hallucinating?

You cannot eliminate hallucination entirely, but you can reduce it to acceptable levels with four practices. First, ground the prompt strictly in retrieved context and instruct the model to refuse if the answer is not present. Second, require citations for each factual claim and surface them to the user. Third, use a faithfulness metric in your evaluation (Ragas does this well) and treat regressions as bugs. Fourth, add a verification step for high-stakes queries where a second model checks the answer against the sources. Together these typically bring hallucination rates from 15-30% on naive systems down to 1-3% on well-engineered ones.

Can we use RAG with our own private data without sending it to OpenAI?

Yes, and this is a common requirement for legal, financial, and healthcare clients. Options include self-hosted open-source models (Llama 3.1, Mistral, Qwen 2.5) served with vLLM or Ollama on your own GPUs; managed enterprise offerings with contractual data-processing terms (Azure OpenAI, AWS Bedrock, Google Vertex AI in UK/EU regions); or private deployments of frontier models via API with zero-retention agreements. Embedding models can similarly be self-hosted. Expect a 10-30% quality gap between the best open models and GPT-4-class systems on complex reasoning, but retrieval-heavy tasks often close that gap because the model's job is mainly to synthesise the retrieved context.

How do we keep the knowledge base current?

The ingestion pipeline needs to be event-driven, not batch-only. Connect to the source systems (SharePoint, Confluence, Salesforce, your CMS) via webhooks or change-data-capture and re-embed documents on change. Delete vectors when source documents are deleted. Version chunks so you can roll back a bad ingestion. Run a nightly reconciliation job that checks the vector store matches the source of truth. For most clients we build this as an n8n workflow feeding a Python ingestion service - the workflow handles source-system quirks, the service handles chunking and embedding consistently.

What team do we need to run a RAG system in production?

Post-launch, a RAG system needs three functions: someone who owns the evaluation set and reviews quality regressions weekly (usually a product manager or domain expert, half a day per week); someone who maintains the ingestion pipelines and vector store (a backend engineer, one to two days per week); and someone who monitors traces and triages user-reported issues (shared with existing on-call). Most mid-market clients cover this with existing staff plus an agency retainer for iteration and upgrades. It is not a full-time headcount unless you are running multiple RAG systems or the corpus is unusually complex.

Where to go next

Start with the smallest useful corpus, the simplest possible pipeline, and a 50-question evaluation set built by whoever owns the domain. Get to a measurable baseline before you add rerankers, hybrid retrieval, or agent loops. Every complexity you add should be justified by a number that moves. If you want to talk through architecture choices for a specific use case, AI Advisory builds RAG systems for UK mid-market clients across professional services, financial services, and B2B SaaS.

Further reading

Sources referenced for context not directly cited in the body:

Ready to put this into production? book a discovery call.

• LangChain Expression Language (LCEL) Overview