Lightweight and perceptually-guided voice conversion for electro-laryngeal speech

2026-01-07

arXiv.org

AI Summary

Electrolaryngeal (EL) speech suffers from poor naturalness and intelligibility due to its constant pitch, limited prosody, and mechanical noise. This work proposes the first adaptation of the lightweight voice conversion framework StreamVC for EL speech rehabilitation, removing pitch and energy modeling modules and instead integrating WavLM features, self-supervised pretraining, and supervised fine-tuning on parallel EL and healthy speech data. The approach further incorporates perceptual loss and human feedback prediction into a joint optimization objective. The proposed method substantially improves speech quality, with the best-performing model (+WavLM+HF) achieving a significant reduction in character error rate and elevating naturalness MOS from 1.1 to 3.3—bringing multiple metrics close to those of healthy human speech.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Electro-laryngeal (EL) speech is characterized by constant pitch, limited prosody, and mechanical noise, reducing naturalness and intelligibility. We propose a lightweight adaptation of the state-of-the-art StreamVC framework to this setting by removing pitch and energy modules and combining self-supervised pretraining with supervised fine-tuning on parallel EL and healthy (HE) speech data, guided by perceptual and intelligibility losses. Objective and subjective evaluations across different loss configurations confirm their influence: the best model variant, based on WavLM features and human-feedback predictions (+WavLM+HF), drastically reduces character error rate (CER) of EL inputs, raises naturalness mean opinion score (nMOS) from 1.1 to 3.3, and consistently narrows the gap to HE ground-truth speech in all evaluated metrics. These findings demonstrate the feasibility of adapting lightweight voice conversion architectures to EL voice rehabilitation while also identifying prosody generation and intelligibility improvements as the main remaining bottlenecks.

Problem

• electro-laryngeal speech
• voice conversion
• naturalness
• intelligibility
• prosody

Innovation

• lightweight voice conversion
• perceptual-guided training
• electro-laryngeal speech
• WavLM features
• human-feedback prediction