What is RAG Fusion? A Practical Guide to Multi-Query Retrieval

Standard retrieval-augmented generation has a known weakness: it depends entirely on the user's original question being a good search query. Most questions are not. They are ambiguous, use different vocabulary from the source documents, or bundle several sub-questions into one sentence. RAG Fusion is a retrieval pattern designed to address exactly this problem by generating multiple query variations and combining their results using reciprocal rank fusion.

The technique was introduced by Adrian Raudaschl in a 2023 write-up on the Towards Data Science publication and has since been adopted widely in production RAG systems. This article explains what RAG Fusion is, how it differs from vanilla RAG and other retrieval patterns, when it is worth the added latency and cost, and how to implement it sensibly.

The problem RAG Fusion solves

Vanilla RAG is a three-step pipeline. Take a user query, embed it, retrieve the top-k most similar chunks from a vector store, and pass those chunks to an LLM as context. It works well when the query is well-formed and shares vocabulary with the corpus. It breaks in three common situations.

First, vocabulary mismatch. A user asks "how do I claim expenses?" but the policy document uses the phrase "submitting reimbursement requests". Dense embeddings help here but do not eliminate the gap, particularly for domain-specific terminology or acronyms.

Second, question ambiguity. "What's our refund policy?" could mean the customer-facing policy, the internal finance process, or the specific rules for a product line. A single embedding averages across all interpretations and often retrieves nothing useful for any of them.

Third, multi-hop or compound questions. "How does our onboarding compare to competitors and what should we change?" contains two distinct information needs. A single retrieval pass typically satisfies one and misses the other.

RAG Fusion attacks all three by generating several reformulations of the original query, running retrieval for each in parallel, and merging the ranked lists into a single set of chunks weighted by how often and how highly each chunk appeared across queries.

How RAG Fusion works

The pipeline has four stages.

Stage 1: query generation. An LLM (usually a cheap, fast model like GPT-4o-mini or Claude Haiku) takes the original user query and produces N variations. Typical N is 3 to 5. The prompt asks for reformulations that preserve intent but use different phrasing, break compound questions into parts, or make implicit terms explicit. For "how do I claim expenses?" you might get: "submitting reimbursement requests process", "employee expense claim procedure", "how to get reimbursed for business expenses", "expense report submission workflow".

Stage 2: parallel retrieval. Each generated query, plus the original, is embedded and used to retrieve the top-k chunks from the vector store. If N=4 variations and k=10, you now have 5 ranked lists of 10 chunks each - up to 50 unique chunks, though there is usually significant overlap.

Stage 3: reciprocal rank fusion (RRF). The ranked lists are merged into a single ranking using the RRF formula from a 2009 paper by Cormack, Clarke and Buettcher. For each unique chunk, its RRF score is the sum across all lists of 1/(k + rank), where rank is its position in that list and k is a constant (usually 60). Chunks that appear high in multiple lists get the highest scores. Chunks that appear in only one list still contribute but are weighted less.

Stage 4: generation. The top-m chunks by RRF score (often m=5 to 10) are passed to the answering LLM alongside the original user query. The variations are discarded - they were only ever a retrieval device.

The elegance of RRF is that it does not need normalised scores across retrievers. It only cares about rank order. This makes it robust to different similarity metrics and retrieval methods, which is why it also underpins hybrid search (combining dense and sparse retrievers).

RAG Fusion vs other retrieval patterns

Several patterns compete in the same space. Choosing between them depends on your latency budget, cost tolerance, and the specific failure modes you see in evaluation.

HyDE (Hypothetical Document Embeddings). Instead of embedding the user query, HyDE asks an LLM to write a hypothetical answer to the query, then embeds that answer and retrieves against it. The intuition is that a fake answer shares more vocabulary with real answers than a question does. HyDE is a single retrieval pass so it is cheaper than RAG Fusion but less robust to ambiguity - one hallucinated answer can send retrieval in the wrong direction.

Multi-Query Retrieval (LangChain's MultiQueryRetriever). This is essentially RAG Fusion without the RRF step - it generates N queries, retrieves for each, then deduplicates the union. Simpler, but you lose the ranking signal that RRF provides. Chunks are treated as equal regardless of how many query variations found them.

Hybrid search. Combines dense (vector) retrieval with sparse (BM25) retrieval, usually fused with RRF. Solves vocabulary mismatch differently - BM25 catches exact term matches that embeddings miss. Hybrid search and RAG Fusion are complementary, not competing. Many production systems use both: generate query variations, run each through a hybrid retriever, then RRF across everything.

Query decomposition. For genuinely compound questions, an LLM splits the query into sub-questions, retrieves and answers each independently, then synthesises. More expensive than RAG Fusion and slower, but better for complex analytical queries where sub-answers need reasoning between them.

Reranking with a cross-encoder. Retrieve a larger candidate set (say 50 chunks) with a fast method, then rerank with a cross-encoder model like Cohere Rerank or bge-reranker. This is often more effective than RAG Fusion for precision and is architecturally simpler. If you can only add one thing to vanilla RAG, a reranker usually beats query fusion on standard benchmarks. The two combine well.

When RAG Fusion is worth the cost

RAG Fusion has real costs. Generating N query variations adds a full LLM call to every user request, typically 200-500ms and a fraction of a penny per call. Running N+1 retrievals in parallel is cheap on the vector database side but does add load. End-to-end you are looking at 300-800ms of added latency depending on how well you parallelise.

It is worth the cost when:

• Your evaluation shows a meaningful gap between retrieval recall (are the right chunks in the top 50?) and top-k precision (are they in the top 5?). RAG Fusion improves the latter without needing a bigger context window.
• Users ask short, ambiguous questions - typical of chatbot interfaces where people type as they would speak.
• Your corpus uses different vocabulary from your users - regulatory documents, technical manuals, medical or legal content.
• You have already done the cheaper wins: proper chunking, hybrid retrieval, a reranker.

It is not worth the cost when:

• Queries are already well-formed - internal search for engineers, code search, SQL-like question interfaces.
• Latency budget is tight (voice assistants, high-throughput classification pipelines).
• You have not yet measured retrieval quality. Adding RAG Fusion blindly often shows no improvement because the bottleneck was elsewhere - poor chunking, missing metadata filters, or an under-trained embedding model.

The ICO's guidance on AI and data protection is worth reading before deploying any RAG system that touches personal data - the multi-query pattern does not change your obligations but does mean more inference calls, which matters for data processor agreements with model vendors.

Implementing RAG Fusion

A minimal implementation is under 100 lines of Python. The moving parts are: query generation prompt, parallel retrieval, RRF scoring, and the final generation call.

Query generation prompt. Keep it tight. Something like: "Generate 4 alternative phrasings of the following question. Vary vocabulary and specificity but preserve intent. Return one per line, no numbering or preamble. Question: {query}". Use a fast model - GPT-4o-mini, Claude 3.5 Haiku, or a self-hosted Llama 3.1 8B. The quality of variations does not need to be high; diversity matters more than eloquence.

Parallel retrieval. Use asyncio in Python or Promise.all in TypeScript. If you are hitting a managed vector database like Pinecone, Weaviate, or Qdrant, the calls will complete in 20-50ms each, so parallelising N+1 requests keeps total retrieval latency close to a single call. Cache embeddings for repeated queries within a session.

RRF scoring. The formula is: score(chunk) = sum over all ranked lists of 1/(60 + rank), where rank starts at 1. Deduplicate chunks by ID before scoring - the same chunk appearing in multiple lists is the entire point. The constant 60 is from the original Cormack paper and works well in practice; do not tune it unless you have a good reason.

Generation. Pass the top 5-10 fused chunks to the answering model with the original user query. Do not pass the query variations - they are noise at this stage. Include chunk metadata (source document, section) in the context so the model can cite properly.

Evaluation. Do not skip this. Build a set of 50-200 real user queries with known correct answers, then measure retrieval recall@k and answer quality with and without RAG Fusion. Ragas and TruLens are the two evaluation frameworks worth trying. If the improvement is under 5 percentage points on your metric of choice, the added latency probably is not justified.

LangChain and LlamaIndex both ship RAG Fusion implementations, though rolling your own is simple enough and gives you more control. The LangChain version is called EnsembleRetriever combined with MultiQueryRetriever; LlamaIndex exposes it through QueryFusionRetriever.

Common failure modes

Three things go wrong in production.

Query drift. The LLM generates variations that are technically related but drift from user intent. "How do I claim expenses?" becomes "tax deductions for business expenses" - superficially similar, semantically different. Fix this with a tighter generation prompt, few-shot examples from your domain, and a lower temperature (0.3 to 0.5 rather than 0.7).

Redundant variations. The model produces four near-identical rephrasings that all retrieve the same chunks. You pay for four LLM calls and get the benefit of one. Fix with explicit diversity instructions in the prompt ("vary vocabulary and specificity") or by post-filtering variations that are too semantically similar to each other before running retrieval.

Fusion of bad results. If the underlying retriever is poor, RRF fuses several bad rankings into one bad ranking. Garbage in, garbage out. RAG Fusion amplifies a good retriever; it does not rescue a broken one. Get the base retrieval right first.

Frequently asked questions

How is RAG Fusion different from just running more retrievals?

Running more retrievals with the same query returns the same chunks - vector search is deterministic for a given query. RAG Fusion generates semantically different queries, so each retrieval explores a different region of the embedding space. The reciprocal rank fusion step then combines the results in a way that rewards chunks appearing across multiple queries, which is a strong signal of relevance. Simply increasing top-k on a single query gets you more chunks but not more diverse chunks, and the marginal chunks are almost always less relevant than the top few.

What is the typical latency overhead?

Expect 300 to 800 milliseconds of added latency end-to-end. The query generation call is the biggest component at 200 to 500ms depending on model choice - Claude Haiku and GPT-4o-mini are at the fast end. Parallel retrieval adds only 20 to 50ms if you async the calls properly. RRF scoring is microseconds. If your baseline RAG response is 2 seconds, RAG Fusion pushes it to 2.5 to 2.8 seconds. For interactive chatbots this is usually acceptable; for voice or high-throughput classification it often is not.

Does RAG Fusion work with hybrid search?

Yes, and the combination is stronger than either alone. Generate N query variations, run each through a hybrid retriever (dense plus BM25), then RRF across all the resulting lists. You end up fusing 2N ranked lists which is more expensive but tends to give the best retrieval quality on real corpora. Weaviate, Qdrant, and Elasticsearch all support hybrid retrieval natively so the additional engineering work is small. Measure before committing - on some corpora the marginal gain over hybrid-alone is negligible.

How many query variations should I generate?

Three to five is the sweet spot. Below three you do not get enough diversity to justify the overhead. Above five you hit diminishing returns because the variations start repeating each other and you are paying for retrievals that add no new chunks. The right number for your system depends on how varied user queries are - broad public chatbots benefit from more variations, narrow internal tools benefit from fewer. Start with four and adjust based on evaluation results.

Do I need a reranker on top of RAG Fusion?

Often yes. RAG Fusion improves which chunks make it into your candidate pool; a reranker like Cohere Rerank or bge-reranker-large then reorders that pool by fine-grained relevance to the original query. They address different problems - fusion addresses recall (finding the right chunks somewhere in the top 50), reranking addresses precision (getting the best 5 to the top). Most production systems that care about answer quality use both. If you can only add one, a reranker typically gives larger gains on standard benchmarks.

Can smaller open-source models handle the query generation step?

Yes. Query generation is a low-stakes task - you need diverse rephrasings, not perfect ones. Llama 3.1 8B, Mistral 7B, or Qwen 2.5 7B all handle it well when self-hosted through vLLM or Ollama. This matters if you are cost-sensitive or if data residency requirements make sending every user query to a US-based API problematic. Keep the generation prompt tight and provide two or three few-shot examples from your domain - small models benefit from this more than large ones.

How do I evaluate whether RAG Fusion is actually helping?

Build an evaluation set of 50 to 200 real user queries with known correct answers or known correct source chunks. Measure retrieval recall@5 and recall@10 with and without fusion - this tells you if the right chunks are making it into the context. Then measure end-to-end answer quality using an LLM-as-judge approach (Ragas and TruLens automate this) or human review on a subset. If recall@10 improves by less than 5 percentage points and answer quality does not move, the added latency and cost are not justified for your use case.

Is RAG Fusion suitable for GDPR-sensitive applications?

The pattern itself is neutral - it does not change what data you process, only how many inference calls you make against it. If your baseline RAG deployment is GDPR-compliant, adding fusion does not change that. Practical considerations: each user query now generates N+1 model calls, which increases exposure to your LLM vendor. If you have a Data Processing Agreement with OpenAI, Anthropic, or another provider, confirm it covers the query generation calls too. Self-hosting the small model that generates variations is a common mitigation - keep sensitive user text inside your own infrastructure and only send the retrieval-grounded final prompt to the larger model.

Conclusion

RAG Fusion is a proven retrieval pattern that meaningfully improves recall on ambiguous or vocabulary-mismatched queries, at the cost of one extra LLM call and some added latency. It is not a silver bullet - the base retriever still needs to be good, and a cross-encoder reranker often gives larger gains for less engineering. But when you have squeezed the obvious wins out of chunking, hybrid search, and reranking, query fusion is the next reasonable step. Evaluate it against your own corpus rather than trusting benchmark numbers, and be honest about whether the latency overhead fits your product.

If you are building a production RAG system and want a second pair of eyes on retrieval architecture, AI Advisory works with mid-market teams on exactly this class of problem.

Further reading

Sources referenced for context not directly cited in the body:

Ready to put this into production? book a discovery call.

• Advanced RAG Techniques - Pinecone