X-VC: Zero-shot Streaming Voice Conversion in Codec Space

2026-04-14

AI Summary

This work proposes a novel approach for zero-shot streaming voice conversion that achieves high-quality, low-latency cross-lingual voice transfer without requiring fine-tuning on the target speaker, while preserving the linguistic content of the source utterance. The method operates in the latent space of a pretrained neural codec and employs a one-step conversion framework, where a dual-conditioned acoustic transformer fuses frame-level acoustic features from a reference target utterance with latent representations of the source speech. Speaker identity is incorporated via adaptive normalization using utterance-level speaker embeddings. To bridge the gap between training and streaming inference, the model leverages a role-allocation training strategy and a chunk-wise overlapping smoothing mechanism. Experiments demonstrate that the proposed method significantly outperforms baseline systems on Seed-TTS-Eval, achieving state-of-the-art performance in English–Chinese streaming word error rate, speaker similarity (both intra- and cross-lingual), and real-time capability.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Zero-shot voice conversion (VC) aims to convert a source utterance into the voice of an unseen target speaker while preserving its linguistic content. Although recent systems have improved conversion quality, building zero-shot VC systems for interactive scenarios remains challenging because high-fidelity speaker transfer and low-latency streaming inference are difficult to achieve simultaneously. In this work, we present X-VC, a zero-shot streaming VC system that performs one-step conversion in the latent space of a pretrained neural codec. X-VC uses a dual-conditioning acoustic converter that jointly models source codec latents and frame-level acoustic conditions derived from target reference speech, while injecting utterance-level target speaker information through adaptive normalization. To reduce the mismatch between training and inference, we train the model with generated paired data and a role-assignment strategy that combines standard, reconstruction, and reversed modes. For streaming inference, we further adopt a chunkwise inference scheme with overlap smoothing that is aligned with the segment-based training paradigm of the codec. Experiments on Seed-TTS-Eval show that X-VC achieves the best streaming WER in both English and Chinese, strong speaker similarity in same-language and cross-lingual settings, and substantially lower offline real-time factor than the compared baselines. These results suggest that codec-space one-step conversion is a practical approach for building high-quality low-latency zero-shot VC systems. Audio samples are available at https://x-vc.github.io. Our code and checkpoints will also be released.

Problem

• zero-shot voice conversion
• streaming inference
• low-latency
• speaker transfer
• interactive scenarios

Innovation

• zero-shot voice conversion
• streaming inference
• codec latent space
• dual-conditioning acoustic converter
• adaptive normalization