Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech

2026-06-14

AI Summary

This work addresses the scarcity of high-quality code-switched text-audio paired data, which hinders the performance of code-switching automatic speech recognition (ASR). The authors propose a code-switching–guided preference learning framework that, for the first time, incorporates the Code-Mixing Index (CMI) into the text-to-speech (TTS) synthesis process. By leveraging CMI to steer synthetic speech generation and integrating preference learning to explicitly enhance linguistic boundary consistency, the method improves the suitability of synthesized data for ASR training. Fine-tuning Whisper Large with the generated data yields substantial gains on the SEAME corpus, reducing the mixed error rate from 12.1% to 8.9% on DevMAN and from 17.8% to 14.2% on DevSGE.

Abstract

Code-switch (CS) Automatic Speech Recognition (ASR) remains challenging due to limited availability of high quality CS text-speech pairs for training. Although synthetic data augmentation via Text-to-speech (TTS) has been explored, existing CS TTS approaches primarily optimise reconstruction fidelity and do not explicitly enforce language-boundary consistency, thereby limiting their effectiveness for CS ASR augmentation. This paper proposes a code-mixing guided preference-learning framework that steers synthetic speech generation toward improved code-switching fidelity using the Code Mixing Index (CMI). Experiments on the SEAME Mandarin-English conversational corpus demonstrate that the proposed method enhances the utility of synthetic data for ASR fine-tuning. Specifically, when fine-tuning Whisper Large, the proposed approach reduces Mixed Error Rate (MER) from 12.1%/17.8% to 8.9%/14.2% on the DevMAN and DevSGE sets, respectively.

Problem

• Code-Switching
• Automatic Speech Recognition
• Synthetic Data
• Data Scarcity
• Code-Mixing

Innovation

• code-switching
• synthetic speech
• preference learning
• Code Mixing Index
• data augmentation