SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning

2026-04-21

AI Summary

This study addresses the absence of a trustworthy reasoning benchmark for Arabic financial natural language processing tailored to Islamic finance, a gap that hinders the development of compliant intelligent assistants. To bridge this void, the authors present the first systematic integration of Sharia principles with financial domain knowledge, constructing a multitask Arabic reasoning benchmark spanning seven task categories. The benchmark is grounded in real-world regulatory, jurisprudential, and corporate documents, rigorously annotated and validated by domain experts, and accompanied by an instruction-tuning dataset and an open-ended evaluation framework driven by scoring rules. Evaluations across 19 mainstream large language models reveal moderate performance on identification tasks but significant deficiencies in causal reasoning and generation—particularly in event causality. The benchmark, evaluation framework, and fine-tuned models are publicly released, establishing a foundational resource for trustworthy AI research in Islamic finance.

Technology Category

• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

English financial NLP has progressed rapidly through benchmarks for sentiment, document understanding, and financial question answering, while Arabic financial NLP remains comparatively under-explored despite strong practical demand for trustworthy finance and Islamic-finance assistants. We introduce SAHM, a document-grounded benchmark and instruction-tuning dataset for Arabic financial NLP and Shari'ah-compliant reasoning. SAHM contains 14,380 expert-verified instances spanning seven tasks: AAOIFI standards QA, fatwa-based QA/MCQ, accounting and business exams, financial sentiment analysis, extractive summarization, and event-cause reasoning, curated from authentic regulatory, juristic, and corporate sources. We evaluate 19 strong open and proprietary LLMs using task-specific metrics and rubric-based scoring for open-ended outputs, and find that Arabic fluency does not reliably translate to evidence-grounded financial reasoning: models are substantially stronger on recognition-style tasks than on generation and causal reasoning, with the largest gaps on event-cause reasoning. We release the benchmark, evaluation framework, and an instruction-tuned model to support future research on trustworthy Arabic financial NLP.

Problem

• Arabic financial NLP
• Shari'ah-compliant reasoning
• benchmark
• financial question answering
• document-grounded reasoning

Innovation

• Arabic Financial NLP
• Shari'ah-compliant reasoning
• document-grounded benchmark
• instruction-tuning dataset
• evidence-based reasoning