What happens to accuracy when documents change?
When document collections change—new versions, updated knowledge, or shifting terminology—standard RAG systems often fail because they rely on static relevance matching. A 2024 study on vulnerability detection found that LLMs using naive RAG achieved only 0.06–0.14 accuracy when asked to distinguish vulnerable code from similar-but-benign patched code [1]. That means the system was barely better than guessing. The problem is that changing collections introduce subtle differences that simple retrieval misses. The same study showed that adding multi-dimensional knowledge from historical vulnerabilities boosted accuracy by 16–24%, proving that reliability depends on how well the system adapts to the new content, not just on retrieving the latest document [1].
Another 2024 study on biomedical RAG found that even when relevant documents were retrieved, the LLM often failed to use them correctly because irrelevant information in the retrieved set interfered with its judgment [3]. Adding document embeddings aligned with the LLM's semantic space improved accuracy by only 2.3% over naive RAG—a modest gain that shows how hard it is to stay reliable when the collection contains complex, domain-specific content [3]. Together, these studies reveal a reliability gap: changing collections introduce noise that standard retrieval cannot filter out.
How can agentic RAG systems adapt to keep reliability high?
Agentic RAG systems that use multi-step reasoning and dynamic tool selection can adapt to changing collections better than static RAG. A 2025 master's thesis on scientific research (Estimation of Distribution Algorithms) built an agentic RAG that combined semantic-lexical search, structured metadata queries, and academic web search within a reasoning loop [6]. This system achieved high-fidelity context retrieval and strong topical alignment across diverse query types, showing that multiple retrieval paths help the system stay accurate even as the underlying papers change [6]. The key is that the agent can decide which tool to use based on the current query and collection state, rather than relying on a single static index.
A 2025 study on source reliability estimation took a different approach: it cross-checked information across multiple sources to estimate which ones were trustworthy, then prioritized those sources during retrieval [5]. This method consistently outperformed standard RAG in scenarios where source reliability varied, and it scaled efficiently as the number of sources grew [5]. For a changing document collection, this means the system can dynamically demote unreliable or outdated sources and promote newer, more credible ones—a direct way to maintain reliability over time.
What are the known failure modes for agentic RAG reliability?
Even advanced agentic RAG systems have clear failure modes that threaten reliability. A 2026 study on red-teaming multimodal agentic RAG found that these systems are vulnerable to attacks across multiple surfaces—text poisoning, image injection, and tool manipulation—that can corrupt the retrieval process [2]. On text-poisoning benchmarks, 73–84% of attack templates were exact duplicates of known attacks, meaning the system was easily fooled by slightly altered content [2]. This shows that as document collections change, malicious or erroneous content can slip in and degrade reliability, especially if the system lacks novelty constraints to filter out recycled attacks.
Another reliability risk is that agentic RAG systems often rely on LLM-based evaluation to judge their own outputs, which can be circular. The 2025 master's thesis explicitly noted that factual verification remained a challenge because the system depended on LLM-based judgment rather than expert-validated benchmarks [6]. Similarly, a 2024 study on vocal training RAG found that while segmentation and semantic similarity improved accuracy, the system still struggled with hallucination when the knowledge base lacked sufficient domain-specific content [4]. These findings converge on a key limitation: agentic RAG systems are only as reliable as the quality and freshness of their underlying document collection, and they cannot fully self-correct when that collection changes in unexpected ways.
About These Sources
This answer is built on 6 studies (3 peer-reviewed, 3 preprints) — published from 2024 to 2026, 6 from 2024 or later — selected as the most relevant from 7 studies that passed quality screening, drawn from 53 papers retrieved from a database of over 500 million.
Sources used in this answer
Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG
In a 2024 study on vulnerability detection, naive RAG achieved only 0.06–0.14 accuracy on distinguishing vulnerable from patched code, but adding multi-dimensional knowledge improved accuracy by 16–24% [1].
MIRROR: Novelty-Constrained Memory-Guided MCTS Red-Teaming for Agentic RAG
A 2026 red-teaming study found that multimodal agentic RAG systems are vulnerable to text poisoning, image injection, and tool manipulation, with 73–84% of attack templates being exact duplicates of known attacks [2].
Document Embeddings Enhance Biomedical Retrieval-Augmented Generation
A 2024 biomedical RAG study found that adding document embeddings aligned with the LLM's semantic space improved accuracy by only 2.3% over naive RAG, highlighting the difficulty of using retrieved content in specialized domains [4].
RAG for Question-Answering for Vocal Training Based on Domain Knowledge Base
A 2024 study on vocal training RAG demonstrated that segmenting the knowledge base and establishing semantic similarities improved response accuracy and contextual relevance, but hallucination remained a challenge when domain knowledge was sparse [5].
Retrieval-Augmented Generation with Estimation of Source Reliability
A 2025 study on Reliability-Aware RAG (RA-RAG) showed that estimating source reliability by cross-checking information across sources and prioritizing reliable documents consistently outperformed standard RAG in scenarios with heterogeneous source reliability [6].
Agentic retrieval augmented generation for estimation of distribution algorithms
A 2025 master's thesis on agentic RAG for scientific research found that combining semantic-lexical search, structured metadata queries, and academic web search within a reasoning loop achieved high-fidelity context retrieval, but factual verification remained a challenge due to reliance on LLM-based evaluation [7].