Why do RAG evaluation metrics sometimes contradict what users actually prefer?
The cleanest example comes from CERN's AccGPT project, where researchers compared GraphRAG, agentic RAG, and a baseline LLM using both individual metrics (like faithfulness and recall) and direct head-to-head LLM-judge comparisons. GraphRAG won on the individual metrics, but the agentic system won decisively in the direct comparison (52.6% preference rate in a three-way test) [4]. This divergence shows that traditional RAG metrics—which measure retrieval quality or factual correctness—may not predict actual answer quality. The authors explicitly concluded that 'traditional RAG metrics may not predict actual answer quality,' highlighting a critical methodological insight [4].
Supporting evidence comes from a neuro-symbolic fusion framework that evaluated how well a knowledge graph supports the generation pipeline. The authors found that static graph metrics (like coverage and consistency) fail to quantify how effectively a graph supports dynamic generation, and they proposed a query-hit-driven score that penalizes ungrounded generation while rewarding appropriate uncertainty when evidence is missing [2]. This suggests that standard evaluation tasks often miss whether the LLM is truly grounded in the graph, especially when retrieval partially fails.
What do RAG evaluations overlook about the real-world cost of graph-enhanced RAG?
Most RAG evaluation tasks focus on accuracy and retrieval metrics, but they often ignore latency and computational overhead—factors that can make a system impractical in production. In the O-RAN study, cache-enhanced RAG reduced average latency by 35.8% compared to baseline Graph-RAG, while Graph-RAG achieved the highest factual correctness (71.2% vs. 70.5% for RAG and 64.8% for LLM) [1]. The paper highlights that Graph-RAG's higher accuracy comes at a cost: it is slower and more resource-intensive, which is a critical trade-off for real-time applications like 6G networks [1].
Similarly, the CERN study reported clear quality-latency trade-offs: AccGPT responded in 0.92 seconds, GraphRAG in 4.34 seconds, and Agentic in 11.81 seconds on average [4]. If an evaluation only reports accuracy, it would miss that GraphRAG's superior metrics come with a 4.7x latency penalty compared to the baseline. This is a blind spot that can lead to deploying a system that is technically accurate but too slow for users.
How do RAG evaluation tasks fail to detect ungrounded or hallucinated answers?
Standard RAG evaluations often measure retrieval recall or answer relevance, but they may not check whether the generated answer is actually grounded in the retrieved evidence. The neuro-symbolic framework explicitly penalizes ungrounded generation and rewards appropriate uncertainty when evidence is missing, showing that conventional metrics can miss this failure mode [2]. In a clinical setting, a hybrid graph RAG system achieved 100% recall for factual queries and zero safety violations, but the authors emphasized that without structured verification, LLMs can still hallucinate [3].
Another angle comes from the agentic heterogeneous graph RAG study, which identified three limitations of existing RAG systems: fixed retrieval strategies, absence of sufficiency evaluation, and lack of structured verification against graph facts [5]. They proposed a verification agent that checks entity, relation, and attribute correctness before finalizing an answer—something standard evaluation tasks don't typically assess. This suggests that RAG evaluations often focus on whether the right documents were retrieved, but not on whether the final answer is logically consistent with the graph structure.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2025 to 2026, 5 from 2024 or later, 1 in Q1 journals — selected as the most relevant from 13 studies that passed quality screening, drawn from 57 papers retrieved from a database of over 500 million.
Sources used in this answer
Cache-Enhanced RAG and Graph-RAG for O-RAN
In an O-RAN benchmark, cache-enhanced RAG cut latency by 35.8% and raised cache hit rate to 42.5%, while Graph-RAG achieved the highest factual correctness (71.2%) but was slower and more resource-intensive.
A Neuro-Symbolic Fusion Framework for Evaluating Knowledge Graph Support in Graph-RAG
A neuro-symbolic framework showed that static graph metrics fail to capture how well a graph supports generation; a query-hit-driven score tracked graph visibility from 99.20 to 46.50 as retrieval accessibility declined.
Unlocking electronic health records: a hybrid graph RAG approach to safe clinical AI for patient QA.
In a clinical EHR study, a hybrid graph RAG achieved 100% recall for factual queries and zero safety violations, but the authors stressed the need for structured verification to prevent hallucinations.
Advancing CERN's AccGPT: Agentic and Graph-Enhanced RAG for Scientific Knowledge Retrieval
At CERN, GraphRAG scored highest on individual metrics, yet agentic RAG won in direct LLM-judge comparisons (52.6% preference), revealing that traditional RAG metrics may not predict answer quality; latency ranged from 0.92s to 11.81s across pipelines.
Agent-Enhanced Heterogeneous Graph RAG for Academic Question Answering.
An agentic heterogeneous graph RAG outperformed baselines by adding query-aware retrieval, sufficiency-aware reranking, and graph-grounded verification, addressing gaps in standard RAG evaluations.