StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversion

2025-06-03

AI Summary

Traditional voice conversion (VC) methods implicitly model linguistic content, hindering simultaneous fidelity preservation of speaker identity and semantic meaning. To address this, we propose StarVC—the first unified autoregressive VC framework—introducing a novel two-stage “text-prediction-first” architecture: (1) explicit linguistic modeling via ASR-guided discrete tokens, and (2) text-conditioned autoregressive acoustic feature generation, enabling disentangled optimization of semantics and timbre. StarVC is the first VC approach to deeply integrate explicit text modeling with acoustic generation under a joint optimization training strategy. Experiments demonstrate that StarVC significantly outperforms state-of-the-art methods in word error rate (WER) and character error rate (CER), achieves a speaker embedding cosine similarity (SECS) of 0.87, and attains a mean opinion score (MOS) of 4.12—validating the substantial quality improvement enabled by language-aware modeling in VC.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Voice Conversion (VC) modifies speech to match a target speaker while preserving linguistic content. Traditional methods usually extract speaker information directly from speech while neglecting the explicit utilization of linguistic content. Since VC fundamentally involves disentangling speaker identity from linguistic content, leveraging structured semantic features could enhance conversion performance. However, previous attempts to incorporate semantic features into VC have shown limited effectiveness, motivating the integration of explicit text modeling. We propose StarVC, a unified autoregressive VC framework that first predicts text tokens before synthesizing acoustic features. The experiments demonstrate that StarVC outperforms conventional VC methods in preserving both linguistic content (i.e., WER and CER) and speaker characteristics (i.e., SECS and MOS). Audio demo can be found at: https://thuhcsi.github.io/StarVC/.

Problem

• Improving voice conversion by leveraging explicit text modeling
• Enhancing speaker identity and linguistic content preservation
• Unifying text and speech generation in autoregressive framework

Innovation

• Unified autoregressive framework for VC
• Predicts text tokens before synthesis
• Enhances linguistic and speaker preservation