WASIL: In-the-Wild Arabic Spoken Interactions with LLMs

2026-05-09

AI Summary

This work addresses the challenge of intent misclassification in Arabic voice assistants, which often stems from ASR errors, ambiguous user utterances, or non-request content, and is further complicated by the difficulty of distinguishing ASR-induced inaccuracies from inherent unanswerability. To tackle this, the authors introduce WASIL, a large-scale dataset of real-world spoken interactions comprising 8,529 dialogue turns in Modern Standard Arabic and four major dialects, including audio, multiple ASR transcripts, assistant responses, and explicit answerability annotations—14.2% of which are negative feedback instances. The study proposes an innovative, low-cost transcription refinement method based on multi-ASR consensus and establishes a reference-free evaluation framework that disentangles ASR errors from intrinsic unanswerability, enabling fine-grained analysis of the end-to-end speech-to-LLM pipeline.

Abstract

Large Language Models (LLMs) voice assistants are commonly built as cascaded Automatic Speech recognition (ASR) to LLM systems, where recognition errors can distort user intent. Dislikes may also arise from ambiguous, out-of-domain, or non-request turns, making it hard to isolate ASR effects. We release WASIL (it denotes connection or linking in Arabic): in-the-wild Arabic spoken interaction prompts with audio, ASR hypotheses, assistant responses, and explicit like/dislike feedback (8,529 turns; 14.2% dislikes), plus a 2,000-turn test set covering Modern Standard Arabic (MSA) and four major dialects with their labels. We provide low-cost gold transcripts via multi-ASR agreement-guided post-editing and annotate answerability (answerable, ambiguous/needs-clarification, unsupported, not-a-request/noise) to separate intrinsic unanswerability from ASR-induced degradation. Finally, we describe scalable reference-free evaluation of responses from ASR vs. gold transcripts using multi-judge LLM scoring.

Problem

• Arabic spoken interactions
• Automatic Speech Recognition (ASR)
• LLM voice assistants
• user intent distortion
• answerability

Innovation

• in-the-wild spoken interactions
• multi-ASR agreement-guided post-editing
• answerability annotation
• reference-free LLM evaluation
• Arabic dialects