AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models

2025-11-18

AI Summary

Existing Arabic and bilingual large language models achieve high scores on superficial linguistic tasks (e.g., spelling, lexical knowledge) but exhibit significant deficiencies in deep linguistic understanding—particularly in grammar, morphology, and syntax—revealing a critical misalignment between benchmark performance and actual linguistic competence. Method: We introduce ArabEval, the first fully human-annotated diagnostic benchmark for Arabic, covering five dimensions: grammar, morphology, spelling, reading comprehension, and syntax. It comprises 150 expert-crafted multiple-choice items explicitly designed to dissociate surface-level knowledge from deep reasoning. We propose a multidimensional diagnostic evaluation framework and open-source all data and evaluation code. Contribution/Results: Systematic evaluation of 35 state-of-the-art models demonstrates pervasive reliance on memorization and pattern matching, with pronounced failures in syntactic parsing and structural reasoning. ArabEval proves highly sensitive to genuine linguistic capability, establishing its validity as a diagnostic tool for uncovering latent model weaknesses.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Evaluation and Analysis
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present AraLingBench: a fully human annotated benchmark for evaluating the Arabic linguistic competence of large language models (LLMs). The benchmark spans five core categories: grammar, morphology, spelling, reading comprehension, and syntax, through 150 expert-designed multiple choice questions that directly assess structural language understanding. Evaluating 35 Arabic and bilingual LLMs reveals that current models demonstrate strong surface level proficiency but struggle with deeper grammatical and syntactic reasoning. AraLingBench highlights a persistent gap between high scores on knowledge-based benchmarks and true linguistic mastery, showing that many models succeed through memorization or pattern recognition rather than authentic comprehension. By isolating and measuring fundamental linguistic skills, AraLingBench provides a diagnostic framework for developing Arabic LLMs. The full evaluation code is publicly available on GitHub.

Problem

• Evaluating Arabic linguistic capabilities of large language models
• Assessing structural language understanding through grammar and syntax
• Measuring gaps between memorization and true linguistic comprehension

Innovation

• Human-annotated benchmark for Arabic linguistic evaluation
• Tests five core categories through multiple-choice questions
• Diagnostic framework isolating fundamental linguistic skills