Low-Burden Data Augmentation for Dysarthric ASR via Zero-Shot Voice Cloning

2026-06-18

AI Summary

This work addresses the challenge of limited performance in automatic speech recognition (ASR) for dysarthric speech, which stems from data scarcity and high speaker heterogeneity. The study proposes a novel data augmentation approach by leveraging zero-shot voice cloning technology (Higgs Audio V2) to synthesize dysarthric speech, which is then combined with real data to fine-tune the Whisper-medium model. This strategy significantly reduces reliance on large amounts of speaker-specific recordings while enabling cost-effective and scalable training. Experimental results demonstrate that the proposed method achieves a word error rate (WER) of 26.00% on the TORGO test set—comparable to fine-tuning with real data alone—and yields even greater improvements for moderate-to-severe dysarthria cases and on the cross-corpus SAP-1102 dataset, with a relative WER reduction of up to 11.45%.

Abstract

Automatic speech recognition remains unreliable for dysarthric speech due to data scarcity and high inter-speaker variability. While synthetic data can address these gaps, traditional methods often require extensive speaker-specific data, reintroducing the collection bottleneck. We investigate zero-shot voice cloning as a low-burden augmentation strategy, using Higgs Audio V2 to clone speakers in the TORGO dataset. We fine-tune (FT) Whisper-medium on cloned, real, and hybrid data and evaluate on held-out real speech. Compared to the zero-shot (31.62%), Clone FT achieved a competitive 26.00% WER, nearly matching the 24.44% and 25.12% seen with Real and Hybrid FT, respectively. Notably, Clone and Hybrid FT outperform Real FT for moderate-severe speakers. Clone FT achieves the best results (11.45% relative) in cross-corpus evaluation on the SAP-1102. These results suggest that zero-shot cloning provides scalable training data that circumvents the costly data collection bottleneck.

Problem

• dysarthric speech
• data scarcity
• speaker variability
• data augmentation
• automatic speech recognition

Innovation

• zero-shot voice cloning
• dysarthric ASR
• low-burden data augmentation
• speech synthesis
• Whisper fine-tuning