Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speech

2025-06-02

AI Summary

Dysarthric speech—characterized by high inter-speaker variability and slowed articulation—severely degrades automatic speech recognition (ASR) performance. To address this, we propose an unsupervised joint prosodic and acoustic transformation framework that maps dysarthric speech to temporally and spectrally normalized representations approximating healthy speech, thereby enhancing ASR robustness. Our key contributions are: (1) a novel syllable-level prosody modeling mechanism tailored to non-uniform segment durations and weakened rhythmic structure; and (2) an end-to-end, unpaired joint transformation built upon an extended RnV (Rhythm-and-Voice) framework, integrating LF-MMI acoustic modeling with Whisper fine-tuning, and incorporating syllable-level duration normalization and spectral mapping. On the TORGO dataset, the LF-MMI model achieves substantial WER reduction—up to 32.7% for severely dysarthric samples—while Whisper fine-tuning yields marginal gains, underscoring the critical role of explicit prosodic modeling.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Automatic speech recognition (ASR) systems struggle with dysarthric speech due to high inter-speaker variability and slow speaking rates. To address this, we explore dysarthric-to-healthy speech conversion for improved ASR performance. Our approach extends the Rhythm and Voice (RnV) conversion framework by introducing a syllable-based rhythm modeling method suited for dysarthric speech. We assess its impact on ASR by training LF-MMI models and fine-tuning Whisper on converted speech. Experiments on the Torgo corpus reveal that LF-MMI achieves significant word error rate reductions, especially for more severe cases of dysarthria, while fine-tuning Whisper on converted data has minimal effect on its performance. These results highlight the potential of unsupervised rhythm and voice conversion for dysarthric ASR. Code available at: https://github.com/idiap/RnV

Problem

• Improving ASR performance for dysarthric speech
• Converting dysarthric-to-healthy speech rhythm and voice
• Reducing word error rates in severe dysarthria cases

Innovation

• Syllable-based rhythm modeling for dysarthric speech
• Extends Rhythm and Voice conversion framework
• Trains LF-MMI models on converted speech