🤖 AI Summary
This study addresses the limited adaptability of large audio language models to Arabic’s diverse dialects and complex linguistic structures, as well as the lack of unified optimization across generative and discriminative tasks. To this end, we present the first systematic investigation into multitask training of large audio models for Arabic, introducing AraMega-SSum—the first Arabic speech summarization dataset—and proposing two novel strategies based on Qwen2.5-Omni (7B): Task-Progressive Curriculum (TPC) and Aligner-based Diverse Sampling (ADS). Our approach significantly accelerates early-stage convergence, improves F1 scores for paralinguistic attribute recognition, and stabilizes decoding in generative tasks. This work offers an efficient solution for adapting Omni-style models to low-resource, multimodal scenarios involving linguistically complex and underrepresented languages such as Arabic.
📝 Abstract
Audio large language models (LLMs) enable unified speech understanding and generation, yet their adaptation to linguistically complex, dialect-rich settings remains underexplored. This paper presents the first systematic study of multi-task instruction tuning for an Arabic-centric audio LLM, covering a hierarchy of generative tasks (ASR, speech summarization) and discriminative tasks (dialect and emotion identification). To support this study, we introduce AraMega-SSum, a novel dataset for Arabic speech summarization. We fine-tune Qwen2.5-Omni (7B) and propose Task-Progressive Curriculum (TPC) along with Aligner-Based Diverse Sampling (ADS), a strategy that constructs information-dense batches by selecting task- and label-balanced examples. Our results reveal a critical efficiency, robustness trade-off: while ADS accelerates initial convergence and boosts paralinguistic F1-scores, its inherent gradient volatility can destabilize generative decoding under prolonged training. Furthermore, while the TPC stabilizes core acoustic mapping, it often induces negative transfer in downstream tasks. We demonstrate that a Hybrid TPC+ADS Strategy provides an optimal training ``recipe'', first establishing a robust representative foundation before employing diversity-aware refinement to capture fine-grained nuances. These findings offer practical guidance for the efficient adaptation of Omni-models in complex, low-resource multimodal environments.