[CVPR 2026/arXiv] From RAG to Agentic RAG: Bridging the Faithfulness Gap in Islamic Question Answering

TL;DR

In high-stakes domains like Islamic jurisprudence, a simple hallucination isn't just a bug—it's a religious risk. This paper presents ISLAMICFAITHQA, a new bilingual benchmark, and an Agentic RAG framework that transforms retrieval from a passive "one-shot" injection into an active, iterative search process. By forcing models to "plan and verify" using Quranic tools, the authors improved accuracy by up to 27%, proving that for specialized knowledge, retrieval-mediated agency beats raw parameter power.

Problem & Motivation: The Danger of "Fluent Ignorance"

Large Language Models (LLMs) are notoriously confident even when they are wrong. In Islamic QA, where queries often involve complex inheritance laws (Inheritance Law makes up 26.4% of this benchmark) or ritual purity, a model that generates a fluent but unsupported answer can mislead users on fundamental religious obligations.

The authors argue that existing benchmarks (like Multiple Choice Questions) are "leaky"—models can guess correctly without truly understanding the text. Furthermore, standard RAG often fails because complex religious questions require disambiguation that a single retrieval step cannot provide.

Methodology: The Grounding Suite

The researchers didn't just build a model; they built an entire ecosystem for "faithful" AI:

1. SFT & RL Alignment: They curated 25K reasoning pairs and used GSPO (Group Sequence Policy Optimization) to align the model with human-like "expert" judgment, rewarding accuracy and penalized unsupported claims.
2. The Quranic Toolset: Instead of just dumping context into the prompt, the model is given tools: search_quran, read_ayah, and get_surah_info.
3. Agentic Iteration: Unlike standard RAG, which retrieves once and then generates, the Agentic RAG setup follows a "Think -> Search -> Inspect -> Answer" loop.

Figure 1: The end-to-end development workflow from SFT to Agentic RAG.

Experiments & Results: Agency is the Key

The results confirm that while domain-specific SFT and RL help, the real breakthrough comes from the Agentic interface.

• The Baseline Gap: Most off-the-shelf models (Llama-3, Mistral) failed significantly, scoring below 30% on the strict Correct/Incorrect/Abstain protocol.
• The Power of Agency: For the Qwen3-4B model, move from 38.85% (Standard RAG) to 48.90% (Agentic RAG). This indicates that for 10% of cases, the model needed to refine its search to find the correct evidence.
• Bilingual Robustness: One of the most impressive findings was that Agentic RAG nearly erased the performance gap between Arabic and English, as the model anchored its generation to the same canonical Arabic sources regardless of the input language.

Table 1: Quantifying the impact of SFT, RL, and Agentic RAG across different backbones.

Critical Analysis & Conclusion

This work marks a shift from "Knowledge in Weights" to "Knowledge in Tools." By treating the Quran as a verifiable database rather than a training corpus, the authors achieve SOTA performance with relatively small models (4B parameters).

Limits & Looking Forward:

• Citation Laundering: The authors correctly identify a new risk—models might use tools to find a citation that looks relevant but doesn't actually support the claim (citation laundering).
• Interpretive Diversity: The current benchmark focuses on "atomic gold answers," which may brush over legitimate scholarly disagreements (khilāf) between different schools of thought.

The Takeaway: For any developer building AI for "high-consequence" domains (Law, Medicine, Religion), this paper is a blueprint. Stop relying on the model's memory; start giving it the agents and tools to verify its own "thoughts."