SAHM: Decoding the Complexity of Arabic Finance and Shari’ah Reasoning

TL;DR

While English financial NLP has matured significantly, the Arabic-speaking world—managing nearly $5 trillion in sovereign wealth—has lacked a rigorous evaluation framework for its unique financial ecosystem. Enter SAHM, the first benchmark specifically designed to test LLMs on Arabic financial reasoning and Shari’ah compliance. The results are a wake-up call: being fluent in Arabic doesn't mean a model understands a balance sheet or a Fatwa.

The Motivation: A Trillion-Dollar Evaluation Gap

Conventional finance benchmarks (like FinQA or TAT-QA) assume a Western framework of interest-bearing loans and bonds. However, Islamic finance operates on principles of risk-sharing (e.g., Murabaha, Sukuk) and strict Shari’ah compliance.

Existing Arabic LLMs like Jais or Fanar are typically tested on general linguistic tasks. SAHM changes the game by introducing 14,380 expert-verified instances that force models to navigate the intersection of technical accounting, market sentiment, and centuries-old Islamic jurisprudence.

Methodology: Human-Expertise in the Loop

The creation of SAHM wasn't just a simple web-scraping job. The researchers employed a sophisticated hybrid pipeline:

1. OCR & Extraction: Converting over 1,200 pages of AAOIFI (Accounting and Auditing Organization for Islamic Financial Institutions) standards into digital text.
2. Task Diversity: Spanning seven tasks—from Multiple Choice Questions (MCQ) on Accounting to open-ended "Event-Cause Reasoning."
3. Verification: Every instance was audited by native experts to ensure that nuances in Fiqh al-mu’amalat (Islamic commercial jurisprudence) were preserved.

Main Findings: Fluency is Not Reasoning

The most striking insight from the SAHM study is the Recognition-Generation Gap. High-performing models like GPT-5 and Claude-Opus-4.5 can identify the correct ruling in a multiple-choice format (scoring >90%) but struggle to generate a coherent, concise explanation when asked an open-ended question.

The "True IQ Test": Event-Cause Reasoning

The Event-Cause Reasoning task proved the most difficult. It requires models to analyze a financial report and explain why an event occurred (e.g., how a central bank decision influenced Sukuk yields). Performance here showed the widest spread (from 1.89 to 9.84 out of 10), acting as a clear differentiator between "intelligent" reasoning and simple pattern matching.

Domain Adaptation vs. Scale

The paper proves that you don't need 100B+ parameters to excel in a niche domain. By fine-tuning smaller 7B-8B Arabic models (like ALLAM and Jais) on the SAHM dataset, the researchers created SAHM-ALLAM-7B, which actually surpassed GPT-5 in Business and Accounting MCQs.

Error Analysis: Why Models Fail

The authors didn't just look at scores; they looked at why models failed. Interestingly, calculation errors (arithmetic) were almost non-existent (0.3%). Instead, models failed because of:

• Conceptual Confusion (58.5%): Conflating related but distinct financial principles.
• Fabricated Evidence (11.4%): Hallucinating Hadith or regulatory articles to justify a wrong answer.
• Verbosity as Uncertainty: Models frequently wrote 4-6x more text than human experts when they were unsure, using "word salad" as a proxy for expertise.

Conclusion: The Path to Trustworthy Assistants

SAHM provides the blueprint for building specialized AI assistants for the MENA region. It highlights that for high-stakes industries like finance and law, general-purpose LLMs still require "targeted surgical intervention" through domain-specific fine-tuning.

The release of the SAHM benchmark and the adapted models on HuggingFace marks a significant step toward making Arabic financial NLP as robust and trustworthy as its English counterpart.

For more technical details, you can find the SAHM Benchmark Code and Dataset on HuggingFace.