Absher: A Benchmark for Evaluating Large Language Models Understanding of Saudi Dialects

2025-07-14

AI Summary

This study addresses the limited understanding of Saudi Arabic dialects by large language models (LLMs). We introduce Absher, the first systematic benchmark for evaluating LLMs on Saudi Arabic, comprising over 18,000 human-verified multiple-choice questions across six task categories—including semantic understanding, cloze completion, binary judgment, and contextual application—and pioneering evaluation dimensions such as cultural interpretation and regional identification. We evaluate both multilingual and Arabic-specific LLMs, revealing substantial deficiencies in dialectal semantic comprehension, cultural reasoning, and contextual inference. Absher fills a critical gap in Arabic dialect NLP evaluation and underscores the necessity of dialect-aware pretraining and culturally aligned assessment. It provides a reproducible, fine-grained evaluation framework and empirical evidence to support robust, real-world deployment of Arabic NLP systems.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) become increasingly central to Arabic NLP applications, evaluating their understanding of regional dialects and cultural nuances is essential, particularly in linguistically diverse settings like Saudi Arabia. This paper introduces exttt{Absher}, a comprehensive benchmark specifically designed to assess LLMs performance across major Saudi dialects. exttt{Absher} comprises over 18,000 multiple-choice questions spanning six distinct categories: Meaning, True/False, Fill-in-the-Blank, Contextual Usage, Cultural Interpretation, and Location Recognition. These questions are derived from a curated dataset of dialectal words, phrases, and proverbs sourced from various regions of Saudi Arabia. We evaluate several state-of-the-art LLMs, including multilingual and Arabic-specific models. We also provide detailed insights into their capabilities and limitations. Our results reveal notable performance gaps, particularly in tasks requiring cultural inference or contextual understanding. Our findings highlight the urgent need for dialect-aware training and culturally aligned evaluation methodologies to improve LLMs performance in real-world Arabic applications.

Problem

• Evaluating LLMs understanding of Saudi dialects and cultural nuances
• Assessing LLMs performance across major Saudi dialect categories
• Identifying gaps in cultural inference and contextual understanding

Innovation

• Introduces Absher benchmark for Saudi dialects
• Includes 18,000 dialectal multiple-choice questions
• Evaluates LLMs on cultural and contextual tasks