Why a static benchmark can't stay reliable when documents change
A benchmark is essentially a frozen test: it measures how well a RAG system performs on a specific set of questions and a specific document collection at a single point in time. If the underlying documents are updated—new medical guidelines are published, legal statutes are amended, or product manuals are revised—the benchmark's relevance erodes because the 'correct' answers may change or the retrieval context may shift. The CRAG benchmark [2] explicitly acknowledges this by designing its questions to reflect temporal dynamism 'ranging from years to seconds' and commits to maintaining the benchmark over time to serve research communities. Without such maintenance, a benchmark becomes a historical artifact rather than a reliable gauge of current performance.
What the benchmarks themselves reveal about sensitivity to document changes
The MIRAGE medical benchmark [1] conducted large-scale experiments with over 1.8 trillion prompt tokens across 41 combinations of different corpora, retrievers, and LLMs. They found that the combination of various medical corpora and retrievers achieved the best performance, and they discovered a 'lost-in-the-middle' effect where the order of retrieved documents matters. This means that even small changes to the document collection—adding, removing, or reordering documents—can significantly alter a RAG system's accuracy. Similarly, RAGBench [3] covers five industry-specific domains with 100k examples sourced from industry corpora like user manuals, which are frequently updated. The benchmark's design for 'continuous improvement of production applications' implies that static evaluation is insufficient for real-world deployments where documents evolve.
Domain-specific challenges: legal and medical benchmarks show the stakes
In the legal domain, LegalBench-RAG [4] focuses on precise retrieval of minimal text segments from legal documents, because long context windows cost more, induce higher latency, and lead LLMs to hallucinate. Legal documents are frequently amended, and the benchmark's 6,858 query-answer pairs were entirely human-annotated by legal experts from a specific corpus of over 79 million characters. If that corpus changes—new case law, updated statutes—the annotations become outdated. The medical benchmark MIRAGE [1] improved LLM accuracy by up to 18% over chain-of-thought prompting, but that improvement was measured against a fixed set of medical QA datasets. Neither benchmark provides a mechanism for automatically adapting to document changes, so their reliability depends on periodic re-annotation and re-release.
How to keep a domain-specific RAG benchmark reliable over time
The strongest evidence across these studies points to three practical steps. First, choose a benchmark that explicitly commits to maintenance—CRAG [2] states it will be maintained to serve research communities, and its design includes temporal dynamism as a core feature. Second, use benchmarks that provide explainable, actionable metrics like RAGBench's TRACe framework [3], which enables continuous evaluation and feedback for production applications. Third, recognize that no single benchmark is sufficient; the MIRAGE study [1] tested 41 combinations and found that performance varies widely, so you should re-run benchmarks whenever your document collection changes. The CRAG benchmark [2] found that even state-of-the-art RAG solutions only answer 63% of questions without hallucination, and accuracy drops further for facts with higher dynamism—meaning the more your documents change, the less reliable a static benchmark becomes.
About These Sources
This answer is built on 4 studies (2 peer-reviewed, 2 preprints) — published in 2024, 4 from 2024 or later, collectively cited 171 times — selected as the most relevant from 4 studies that passed quality screening, drawn from 53 papers retrieved from a database of over 500 million.
Sources used in this answer
Benchmarking Retrieval-Augmented Generation for Medicine
MIRAGE [1] tested 41 combinations of corpora, retrievers, and LLMs across 7,663 medical questions, finding that accuracy improved up to 18% with RAG but varied significantly with corpus choice, and discovered a 'lost-in-the-middle' effect where document order matters.
CRAG -- Comprehensive RAG Benchmark
CRAG [2] includes 4,409 question-answer pairs across five domains with temporal dynamism from years to seconds, and found that even top RAG solutions only answer 63% of questions without hallucination, with accuracy dropping for more dynamic or less popular facts.
RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
RAGBench [3] provides 100k examples across five industry domains from real user manuals, and its TRACe evaluation framework enables continuous, explainable feedback for production RAG systems.
LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain
LegalBench-RAG [4] contains 6,858 human-annotated query-answer pairs from a legal corpus of over 79 million characters, focusing on precise retrieval of minimal text segments to avoid context window issues and hallucinations.