Dialectal Coverage And Generalization in Arabic Speech Recognition

2024-11-07

arXiv.org

AI Summary

Existing Arabic automatic speech recognition (ASR) systems exhibit strong bias toward Modern Standard Arabic (MSA) and a few high-resource dialects, offering inadequate coverage for 11 low-resource spoken varieties and poor robustness to pervasive Arabic–English and Arabic–French code-switching. Method: We propose the first open-source ASR framework unifying MSA, 11 regional dialects, and bilingual code-switched speech, trained on speech data from 17 countries. Our approach integrates multi-task pretraining, dialect-adaptive fine-tuning, multilingual joint modeling, and geography-aware data fusion. Contribution/Results: On a multi-variant test set, our system reduces dialect-specific word error rates (WER) by 22% on average and lowers WER on code-switched utterances by 18%. It significantly improves generalization to low-resource dialects and cross-variant robustness. All models and curated datasets are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Conversational AI/Dialog Systems
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Developing robust automatic speech recognition (ASR) systems for Arabic requires effective strategies to manage its diversity. Existing ASR systems mainly cover the modern standard Arabic (MSA) variety and few high-resource dialects, but fall short in coverage and generalization across the multitude of spoken variants. Code-switching with English and French is also common in different regions of the Arab world, which challenges the performance of monolingual Arabic models. In this work, we introduce a suite of ASR models optimized to effectively recognize multiple variants of spoken Arabic, including MSA, various dialects, and code-switching. We provide open-source pre-trained models that cover data from 17 Arabic-speaking countries, and fine-tuned MSA and dialectal ASR models that include at least 11 variants, as well as multi-lingual ASR models covering embedded languages in code-switched utterances. We evaluate ASR performance across these spoken varieties and demonstrate both coverage and performance gains compared to prior models.

Problem

• Addressing dialect diversity in Arabic speech recognition
• Improving coverage of Modern Standard Arabic and dialects
• Handling code-switching with English and French

Innovation

• Optimized ASR models for multiple Arabic variants
• Open-source pre-trained models covering 17 countries
• Multilingual ASR for code-switched utterances