Adding Robust Code-Switching Capabilities to High Performance Multilingual ASR

2026-06-20

AI Summary

This work addresses the significant performance degradation of existing multilingual automatic speech recognition (ASR) systems when handling code-switching, a challenge exacerbated by conventional fine-tuning approaches that often compromise strong monolingual capabilities. To overcome this limitation, the authors propose a Bayesian factorized adaptation mechanism that efficiently injects code-switching knowledge into a pretrained multilingual ASR model using only a small amount of synthetic data. This approach enhances the model’s robustness to complex code-switching phenomena—including cross-lingual morphological variations—without overwriting its original competencies. The method substantially outperforms strategies relying on large-scale real-world data, achieving a 32.87% relative reduction in code-switching word error rate (WER) and a 5.31% absolute improvement in overall WER, while fully preserving monolingual recognition performance.

Abstract

Code-switching (CSW) remains challenging for large multi-lingual ASR systems in real-world deployment. While fine-tuning on synthetic CSW data is possible, it generally degrades strong monolingual baselines. Our goal is to preserve these capabilities while extending models to handle complex code-switching, including morphological variations across languages. We propose Bayesian factorized adaptation, which learns to efficiently integrate switching-relevant knowledge into strong pretrained models without overwriting existing capabilities. Requiring only a small amount of synthetic data, our approach reduces transcription errors by 32.87% on code-switched words while improving overall WER by 5.31%, all while maintaining mono-lingual performance. Our results demonstrate that effective CSW adaptation depends more on knowledge integration than data complexity.

Problem

• code-switching
• multilingual ASR
• robustness
• morphological variation
• monolingual performance

Innovation

• Bayesian factorized adaptation
• code-switching
• multilingual ASR
• knowledge integration
• synthetic data