Why do RAG systems so often retrieve the wrong evidence?
The core problem is that current retrieval methods are not good enough at finding the specific, relevant pieces of evidence a user needs, especially when the question is complex or the information is obscure. In a medical question-answering study using the MedQuAD dataset, the RAG system's retrieval precision was only 0.15, meaning that out of every 20 documents it pulled, only 3 were actually relevant to the question [5]. This low retrieval quality directly caused the RAG-enhanced models to perform worse than the same models used without any retrieval at all [5].
The challenge is even steeper for questions that require connecting multiple pieces of information. A dedicated benchmark for multi-hop queries found that existing RAG systems are 'unsatisfactory' at both retrieving and answering such questions, because they need to find and reason over several separate documents simultaneously [1]. This means that for a user asking a question like 'What was the population of the city where the 2012 Olympics were held?', the system often retrieves a document about the Olympics but misses the one with the population data.
When does retrieval fail most dramatically?
Retrieval is particularly weak for questions about rapidly changing facts, niche topics, or tasks that go beyond simple question-answering. The Comprehensive RAG Benchmark (CRAG) showed that accuracy drops sharply for questions about facts with 'higher dynamism' (e.g., current stock prices vs. historical events), 'lower popularity' (long-tail entities), and 'higher complexity' [3]. For instance, a question about a breaking news event might fail because the knowledge base hasn't been updated, while a question about an obscure scientific term might fail because there are few relevant documents to retrieve.
Beyond just answering questions, RAG is also used for tasks like creating, updating, or summarizing content. A comprehensive Chinese benchmark (CRUD-RAG) found that evaluating RAG only on question-answering overlooks these other critical scenarios, where the retrieval component can have a very different impact [2]. For example, when a user wants to 'update' a text with the latest information, the system might retrieve outdated or contradictory evidence, leading to a worse result than if it had used no external knowledge at all.
Does better retrieval solve the hallucination problem?
Even when the right evidence is retrieved, RAG systems still frequently produce hallucinations—confidently stated but incorrect information. In the CRAG benchmark, the most advanced industry RAG solutions still hallucinated on 37% of questions [3]. This shows that retrieval is only one part of the puzzle; the large language model (LLM) itself often fails to correctly use the evidence it receives.
A new benchmark for Russian-language RAG (RusHallu-RAG) identified six distinct types of hallucinations, including 'Contradiction' (the answer contradicts the retrieved document) and 'Missing' (the answer includes information not in the document) [4]. This fine-grained analysis reveals that even when the evidence is perfect, the LLM can misinterpret, ignore, or embellish it. The study also found that larger models are not necessarily better at detecting their own hallucinations, with some medium-sized models (20-33 billion parameters) outperforming larger ones [4].
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2024 to 2026, 5 from 2024 or later, 1 in Q1 journals, collectively cited 258 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 65 papers retrieved from a database of over 500 million.
Sources used in this answer
MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
This paper created the MultiHop-RAG benchmark and found that existing RAG systems perform unsatisfactorily on multi-hop queries, which require retrieving and reasoning over multiple pieces of evidence [1].
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
This paper introduced the CRUD-RAG benchmark, which categorizes RAG applications into Create, Read, Update, and Delete, and found that most existing benchmarks neglect non-question-answering scenarios where RAG's impact differs [2].
CRAG -- Comprehensive RAG Benchmark
This paper presented the CRAG benchmark of 4,409 questions and found that even state-of-the-art industry RAG solutions only answer 63% of questions without hallucination, with accuracy dropping for dynamic, obscure, or complex facts [3].
RusHallu-RAG: benchmarking hallucination detection for Russian RAG
This paper introduced RusHallu-RAG, a benchmark for detecting hallucinations in Russian RAG, and found that model scale does not guarantee performance, with some medium-sized models (20B-33B parameters) outperforming larger ones [4].
Does Domain-Specific Retrieval Augmented Generation Help LLMs Answer Consumer Health Questions?
This paper evaluated RAG on consumer health questions and found that vanilla LLMs consistently outperformed RAG-enhanced approaches, partly due to low retrieval performance (Precision@5 = 0.15) [5].