EDRAC: Benchmarking Arabic Dialect Reading Comprehension

2026-09-01

AI Summary

本文通过构建EDRAC基准，解决方言阿拉伯语阅读理解资源不足的问题，采用人机协作生成499段落及4,977个问答对。

Abstract

Dialectal Arabic (DA) remains under-resourced compared to Modern Standard Arabic (MSA), particularly for machine reading comprehension (MRC) and question answering (QA). Existing Arabic QA benchmarks primarily focus on formal written MSA or multiple-choice QA, with limited coverage of naturally spoken dialects. Here, we aim to bridge this gap. We introduce EDRAC, the first large-scale benchmark for dialectal Arabic machine reading comprehension (MRC) and generative QA, covering five major dialects: Egyptian, Moroccan, Emirati, Syrian, and Saudi Arabic. EDRAC contains 499 passages derived from naturally occurring spoken interactions and 4,977 corresponding QA pairs generated through a human--LLM collaborative pipeline combining iterative generation, LLM-as-a-judge evaluation, and human verification. We benchmark Arabic-centric and multilingual LLMs on EDRAC using lexical and semantic metrics. Our results reveal substantial gaps between semantic answer quality and dialectal fidelity, highlighting the limitations of existing evaluation metrics for dialectal Arabic generation. EDRAC provides a realistic and challenging MRC benchmark for future research on dialectal Arabic NLP.

Problem

• Dialectal Arabic
• Machine Reading Comprehension
• Question Answering
• Benchmark
• Resource Underdevelopment

Innovation

• dialectal Arabic
• machine reading comprehension
• generative QA
• LLM-as-a-judge
• human--LLM collaboration