Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization

2026-06-18

AI Summary

This work addresses the challenge in streaming zero-shot voice conversion of disentangling speaker identity from linguistic content, where existing approaches struggle to simultaneously suppress speaker leakage and preserve vocal expressiveness. The study introduces speaker anonymization into this task for the first time, proposing a novel perturbation mechanism that explicitly protects speaker identity while retaining prosodic information. Furthermore, it designs a strictly causal, non-lookahead generative network to enable truly zero-latency streaming conversion. Without relying on future-frame buffering, the method effectively balances speaker privacy and speech utility, significantly enhancing both the naturalness and real-time performance of converted speech.

Abstract

Streaming zero-shot voice conversion struggles to disentangle timbre from linguistic content without degrading utility or inflating latency. Current methods rely on information bottleneck (IB) or speaker perturbation. While IB filters out timbre, it discards prosody, forcing models to explicitly inject features like fundamental frequency. This often requires buffering future frames, creating algorithmic lookahead latency. On the other hand, existing perturbation methods largely overlook the crucial trade-off between timbre leakage and utility preservation. Recognizing this neglected trade-off, we find that the inherent objective of Speaker Anonymization (SA) aligns well with balancing these factors. Thus, we introduce SA as a novel perturbation mechanism to explicitly mitigate timbre leakage while retaining prosodic utility. Crucially, SA's robust representations significantly alleviate the generator's reliance on future context, enabling our strictly causal, zero-lookahead network. Audio samples are available at https://amphionteam.github.io/Zero-VC-demo/.

Problem

• streaming voice conversion
• timbre disentanglement
• zero-shot
• lookahead latency
• speaker anonymization

Innovation

• Zero-shot voice conversion
• Speaker Anonymization
• Zero-lookahead streaming
• Timbre disentanglement
• Prosody preservation