GenVC: Self-Supervised Zero-Shot Voice Conversion

2025-02-06

AI Summary

Existing zero-shot voice conversion (VC) methods rely on external supervised models to disentangle content and speaker characteristics and typically employ parallel synthesis, limiting speaker similarity and privacy preservation. To address these limitations, GenVC introduces the first fully self-supervised zero-shot VC framework. It achieves unsupervised content–style disentanglement via phoneme-guided self-supervised representation learning—eliminating the need for pre-trained speech recognizers—and adopts an autoregressive generative architecture enabling time-flexible non-parallel synthesis, thereby fully decoupling from source prosody and speaker identity. Without requiring any parallel data or external models, GenVC achieves state-of-the-art speaker similarity, matches mainstream methods in naturalness, and significantly enhances voice anonymization capability and privacy protection.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Security and Privacy: Privacy-enhancing technologies

Abstract

Zero-shot voice conversion has recently made substantial progress, but many models still depend on external supervised systems to disentangle speaker identity and linguistic content. Furthermore, current methods often use parallel conversion, where the converted speech inherits the source utterance's temporal structure, restricting speaker similarity and privacy. To overcome these limitations, we introduce GenVC, a generative zero-shot voice conversion model. GenVC learns to disentangle linguistic content and speaker style in a self-supervised manner, eliminating the need for external models and enabling efficient training on large, unlabeled datasets. Experimental results show that GenVC achieves state-of-the-art speaker similarity while maintaining naturalness competitive with leading approaches. Its autoregressive generation also allows the converted speech to deviate from the source utterance's temporal structure. This feature makes GenVC highly effective for voice anonymization, as it minimizes the preservation of source prosody and speaker characteristics, enhancing privacy protection.

Problem

• Zero-shot voice conversion
• Self-supervised learning
• Voice anonymization

Innovation

• Self-supervised linguistic content disentanglement
• Autoregressive generation for temporal deviation
• Enhanced privacy via speaker anonymization