From RAG to Agentic RAG: Achieving Faithfulness in Islamic Question Answering

TL;DR

Large Language Models (LLMs) are notoriously prone to "confident hallucinations," a risk that becomes unacceptable when the topic is religious guidance. This paper introduces ISLAMICFAITHQA, a rigorous generative benchmark designed to test if models can ground themselves in canonical sources. The authors demonstrate that while standard retrieval (RAG) helps, an Agentic RAG approach—where the model behaves like a researcher, iteratively searching and verifying Quranic verses—yields a massive performance leap, even for smaller 4B-parameter models.

The "Hallucination" Trap in Faith-Based AI

In the Islamic domain, a wrong answer isn't just a factual error; it's a potential misrepresentation of divine law (Sharia). Current models often sound fluent and authoritative while being completely ungrounded.

The paper identifies three critical gaps in current Islamic AI research:

1. Evaluation Proxy Failure: Most benchmarks use Multiple Choice Questions (MCQ), which allow models to "guess" without understanding.
2. Language Asymmetry: Many models perform well in English but fail in Arabic (or vice-versa), leading to inconsistent guidance.
3. The Abstention Problem: Models are too "eager to please" and rarely admit when they don't have enough evidence to provide an Islamic ruling.

Methodology: The Agentic Grounding Suite

To solve this, the researchers didn't just build a dataset; they built an entire ecology for grounded reasoning.

1. The Resource Trinity

• SFT Reasoning (25K pairs): Training models to "think" through theological steps rather than just spitting out facts.
• RL Preference (5K samples): Aligning models using a specialized reward signal (GSPO) to discourage unsupported claims.
• Quran Retrieval Corpus: A verse-by-verse (Ayat) index of ~6,000 units with standardized metadata for precise citations.

2. The Move to Agentic RAG

The "Agentic" part is the secret sauce. Instead of the system simply grabbing the top 5 verses and feeding them to the LLM (Standard RAG), the LLM is given a toolset. It can:

• Perform a semantic search.
• Read a specific verse.
• Look up Surah metadata.
• Refine its search if the first results are ambiguous.

Figure: The end-to-end development workflow, from supervised fine-tuning to tool-mediated agentic inference.

Experiments and Results

The benchmark results are sobering for general-purpose LLMs. Most off-the-shelf models (including Llama 3.1 and Mistral) score below 30% accuracy under strict grading.

However, the transition to Agentic RAG shows explosive growth:

• Backbone Power: While the specialized Fanar-2-27B leads the pack, the application of Agentic RAG brought the Qwen3-4B model from a mediocre 21.85% to a highly competitive 48.90%.
• Bilingual Robustness: Agentic RAG nearly erased the "language gap." Because the model is forced to inspect the same canonical Arabic verses regardless of whether the query was in English or Arabic, the results became much more consistent across languages.

Table: The impact of SFT, RL, RAG, and Agentic RAG across different backbones.

Critical Insight: Why Does It Work?

The paper’s success rests on the "System 2" reasoning approach. Standard RAG is a "System 1" process—fast, intuitive, but often shallow. Agentic RAG forces the model into a deliberate "System 2" loop: Plan $\to$ Search $\to$ Inspect $\to$ Verify $\to$ Answer. This loop is essential for Islamic Jurisprudence (Fiqh), where a single word in a verse can change a ruling's entire context.

Conclusion and Limitations

The authors have provided a roadmap for building trustworthy AI in sensitive, high-stakes domains. By shifting the focus from "scaling parameters" to "scaling verification," they achieve massive gains with modest hardware.

Limitations to consider:

• Quran-Centric: The current RAG index focuses on the Quran. Future work must bridge this with the Hadith (Prophetic traditions) and the complexities of various schools of thought (Madhhabs).
• Latency: Agentic reasoning takes more time (multiple tool calls), which might impact real-time user experience.

Ultimately, this work proves that for religious guidance, the model's ability to search and cite is more important than its ability to memorize.