MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidances

2025-09-21

AI Summary

This paper addresses zero-shot voice conversion (VC) with fine-grained, multi-factor controllability over speaker identity, linguistic content, and prosody. Methodologically, it introduces the first unified masked speech encoder-decoder Transformer architecture incorporating multi-path classifier-free guidance (CFG), enabling joint conditional modeling of continuous/discrete linguistic features, pitch contours, and accent attributes. It further proposes a hybrid linguistic representation scheme that fuses quantized and continuous language embeddings, optionally augmented with pitch-guided conditioning to improve prosodic control fidelity. Experimental results demonstrate that the proposed model significantly outperforms existing baselines in target speaker similarity and accent matching, while achieving word/character error rates comparable to the best-performing baseline—thereby jointly attaining high naturalness and precise, interpretable controllability across speaker, linguistic, and prosodic dimensions.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We introduce MaskVCT, a zero-shot voice conversion (VC) model that offers multi-factor controllability through multiple classifier-free guidances (CFGs). While previous VC models rely on a fixed conditioning scheme, MaskVCT integrates diverse conditions in a single model. To further enhance robustness and control, the model can leverage continuous or quantized linguistic features to enhance intellgibility and speaker similarity, and can use or omit pitch contour to control prosody. These choices allow users to seamlessly balance speaker identity, linguistic content, and prosodic factors in a zero-shot VC setting. Extensive experiments demonstrate that MaskVCT achieves the best target speaker and accent similarities while obtaining competitive word and character error rates compared to existing baselines. Audio samples are available at https://maskvct.github.io/.

Problem

• Achieving zero-shot voice conversion with multi-factor controllability
• Balancing speaker identity, linguistic content, and prosodic factors
• Enhancing intelligibility and speaker similarity in voice conversion

Innovation

• Masked Voice Codec Transformer for zero-shot conversion
• Multiple classifier-free guidances for multi-factor controllability
• Leverages continuous or quantized linguistic and pitch features