DisSpeech: Low-Resource Controllable Mandarin Stuttered Speech Synthesis for ASR Augmentation

2026-06-19

AI Summary

This work addresses the challenge of insufficient robustness in Chinese stuttered speech recognition due to data scarcity and the difficulty of modeling disfluencies such as repetitions, prolongations, and blocks. To this end, the authors propose DisSpeech, a framework based on discrete speech tokens that enables controllable stuttered speech synthesis under low-resource conditions. By incorporating explicit stutter event labels to govern disfluency types and employing a non-autoregressive masked generative Transformer to map text and event labels into semantic speech tokens, the method explicitly models pitch and energy for prosody-aware acoustic reconstruction. With less than 50 hours of stuttered speech for fine-tuning, DisSpeech generates high-quality, controllable stuttered utterances. Augmenting training data with these synthetic samples substantially improves the performance of various ASR systems, achieving a state-of-the-art character error rate of 4.19% on Chinese stuttered speech recognition with Qwen3-ASR-0.6B while minimally affecting recognition accuracy on fluent speech.

Abstract

Stuttered speech recognition remains challenging, with disfluencies such as repetitions, prolongations, and blocks disrupting speech continuity and acoustic patterns. This problem is further aggravated in Mandarin scenarios by the limited availability of stuttered speech data, which makes it difficult to train robust ASR models for diverse disfluency patterns. To address this problem, this paper proposes DisSpeech, a discrete speech token-based framework for low-resource controllable Mandarin stuttered speech synthesis and ASR data augmentation. The proposed framework introduces explicit stuttering event labels to control different disfluency patterns. Text and stuttering event labels are mapped into semantic speech tokens by a non-autoregressive masked generative Transformer, followed by prosody-aware acoustic reconstruction with explicit pitch and energy modeling. With fine-tuning using less than 50 hours of Mandarin stuttered speech, DisSpeech can generate controllable stuttered speech with competitive speech quality. Experimental results show that the proposed method outperforms previous stuttered speech synthesis methods in both speech quality and event controllability. Furthermore, the synthesized stuttered speech effectively improves multiple ASR models, with Qwen3-ASR-0.6B achieving a state-of-the-art CER of 4.19% on the evaluated Mandarin stuttered speech recognition task, while causing only slight degradation on fluent speech.

Problem

• stuttered speech recognition
• low-resource
• Mandarin
• disfluency
• ASR augmentation

Innovation

• stuttered speech synthesis
• discrete speech tokens
• controllable disfluency
• low-resource ASR augmentation
• non-autoregressive Transformer