Why relevance alone doesn't mean the evidence is useful
The core problem is that standard retrieval systems rank documents by how topically similar they are to the query, but that doesn't guarantee the document actually helps answer the question. A 2025 study on utility-driven reranking found that conventional relevance-based retrieval often misaligns with utility — meaning a document can be on-topic yet useless for generating a correct answer [2]. This gap is measurable: a 2024 evaluation study showed that judging retrieval quality by query-document relevance labels has only a small correlation with the RAG system's downstream performance [3]. In other words, a document that looks relevant to a human annotator may not help the language model produce a better answer.
Reranking can dramatically improve what the model actually uses
Reranking pipelines that reorder initial retrieval results based on utility or reasoning structure can substantially close the gap between relevance and usefulness. The LiR³AG framework, for example, restructures retrieved evidence into coherent reasoning chains, improving an 8-billion-parameter model's F1 score by 6.2% to 22.5% across tasks — enough to surpass a much larger 32-billion-parameter reasoning model [1]. Similarly, the LURE-RAG framework uses a listwise ranking loss guided by LLM utility, achieving 97-98% of the performance of state-of-the-art dense neural baselines while being far more efficient, and its dense variant outperforms the best existing baseline by up to 3% [2]. These gains show that reranking doesn't just reorder documents — it can fundamentally change whether the model gets the evidence it needs.
The catch: reranking isn't a silver bullet and can introduce new problems
Even with reranking, the system can still retrieve irrelevant or unhelpful passages. The Self-RAG framework (2023) found that indiscriminately retrieving and incorporating a fixed number of passages — even after reranking — can diminish LM versatility or lead to unhelpful responses [5]. Their solution was to train the model to adaptively retrieve passages on-demand and self-reflect using special 'reflection tokens,' which significantly outperformed standard retrieval-augmented models on fact verification and long-form generation [5]. This suggests that reranking alone isn't enough; the model also needs to decide whether to retrieve at all and whether the retrieved evidence is actually trustworthy. Additionally, a 2026 study noted that reasoning models in RAG often introduce substantial computational costs, including increased token consumption and inference latency — though their lightweight reranking strategy cut output tokens by 98% and inference time by 58.6% [1], showing that efficiency gains are possible but require careful design.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2023 to 2026, 4 from 2024 or later, collectively cited 1,546 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 53 papers retrieved from a database of over 500 million.
Sources used in this answer
LiR3AG: A Lightweight Rerank Reasoning Strategy Framework for Retrieval-Augmented Generation
LiR³AG, a lightweight reranking framework that restructures retrieved evidence into reasoning chains, improved an 8B model's F1 by 6.2% to 22.5% across tasks, surpassing a 32B reasoning model while cutting output tokens by 98% and inference time by 58.6%.
LURE-RAG: Lightweight Utility-driven Reranking for Efficient RAG
LURE-RAG, a utility-driven reranker using LambdaMART with listwise ranking loss, achieved 97-98% of state-of-the-art dense neural baseline performance, and its dense variant outperformed the best existing baseline by up to 3%.
Evaluating Retrieval Quality in Retrieval-Augmented Generation
eRAG, a new evaluation approach, showed that traditional query-document relevance labels have only a small correlation with downstream RAG performance, while eRAG achieved higher correlation (Kendall's tau improvements of 0.168 to 0.494) and used up to 50x less GPU memory.
BKRAG : A BGE Reranker RAG for similarity analysis of power project requirements
BKRAG, which uses a BGE reranker to re-rank candidate documents for semantic matching, significantly improved retrieval accuracy and similarity analysis for power project requirements in a domain-specific application.
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Self-RAG, which adaptively retrieves passages on-demand and uses self-reflection tokens, significantly outperformed ChatGPT and retrieval-augmented Llama2-chat on open-domain QA, reasoning, and fact verification, and improved factuality and citation accuracy for long-form generations.