🤖 AI Summary
This work addresses the challenge of cross-lingual optimization conflicts in multilingual large language model-based speech recognition, where joint training struggles to preserve language-specific characteristics. The authors propose a language-specific multi-teacher online policy distillation framework that integrates language routing with token-level knowledge fusion. To decouple language-specialized capabilities from general multilingual modeling, they introduce both static and dynamic acoustic prefix designs. Evaluated on a mixed benchmark comprising Mandarin, Chinese dialects, Cantonese, and English, the proposed method significantly outperforms reinforcement learning baselines and consistently surpasses all monolingual teacher models, demonstrating superior generalization performance.
📝 Abstract
Modern LLM-based ASR systems have established multilingual capability as a standard feature, leveraging large-scale multilingual corpora and LLMs' cross-lingual knowledge to achieve competitive performance across multilingual benchmarks. However, joint modeling of languages with heterogeneous acoustic, phonological, and lexical characteristics inevitably introduces optimization conflicts, undermining language-wise specialization. To address this challenge, we propose Language-Specialized Multi-Teacher On-Policy Distillation (LS-MOPD), which decouples language-specific knowledge acquisition from multilingual capability integration: language-specialized teachers are independently optimized via reinforcement learning (RL), after which their expertise is integrated into a generalist multilingual student through language routing and token-level multi-teacher distillation, thereby reducing direct cross-lingual optimization conflicts. We further explore two acoustic-prefix configurations, static and dynamic, to examine how teacher--student prefix consistency influences the efficacy of on-policy distillation. Experiments on benchmarks covering Mandarin, Mandarin subdialects, Cantonese, and English demonstrate that LS-MOPD substantially outperforms RL baselines and consistently surpasses the empirical performance envelope defined by best-performing RL teachers, revealing its potential to generalize beyond all teachers in multilingual ASR.