Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization

2026-08-12

AI Summary

Traditional speech semantic tokenizers, being decoupled from acoustic generation tasks, struggle to preserve speaker timbre details, limiting the naturalness and speaker similarity of zero-shot text-to-speech (TTS) systems. This work proposes Phoenix TTS, a novel framework that, for the first time, integrates flow matching–driven joint training into speech tokenization, enabling discrete tokens to retain semantic content while aligning with the continuous acoustic space of downstream generative models. By unifying semantic and acoustic modeling within a single architecture, the approach supports high-quality zero-shot voice conversion without requiring fine-tuning. Trained on 110,000 hours of speech data, the system achieves synthesized speech with word error rates lower than those of real recordings and demonstrates speaker similarity comparable to or exceeding that of leading large models.

Abstract

In current zero-shot text-to-speech systems, conventional semantic tokenizers are typically optimized using supervised automatic speech recognition or self-supervised learning objectives. However, due to the inherent nature of speech, semantic and acoustic information cannot be completely decoupled, and ASR-based tokenizers discard acoustic details to focus on linguistic content; models relying on them usually struggle to achieve optimal speaker similarity. Furthermore, these tokenizers are optimized independently and lack direct supervision from downstream acoustic generation tasks. This isolated training creates a feature gap between the extracted discrete tokens and the continuous space required by acoustic models, fundamentally bottlenecking the upper bound of synthesis quality. To bridge this gap, we propose Phoenix TTS, a unified framework that tightly couples representation learning with generative acoustic modeling. Specifically, our speech tokenizer is optimized to reconstruct self-supervised features to maintain semantic richness, while simultaneously receiving direct supervision from a Flow Matching training loss. Through this joint training paradigm, the extracted discrete tokens successfully preserve essential semantic information and natively align with the feature space of the downstream Flow Matching model. Comprehensive evaluations highlight the efficiency and effectiveness of Phoenix TTS. Trained on 110K hours of data, the system achieves excellent speech intelligibility, yielding WER that consistently falls below that of ground-truth recordings. Simultaneously, it maintains robust zero-shot speaker similarity, rivaling or outperforming several prominent large-scale baselines. Furthermore, as an advantageous byproduct of this unified training, the learned tokenizer can be seamlessly adapted to zero-shot voice conversion tasks without requiring task-specific fine-tuning.

Problem

• text-to-speech
• speech tokenization
• zero-shot speaker similarity
• acoustic modeling
• feature alignment

Innovation

• Flow Matching
• Speech Tokenization
• Zero-shot TTS
• Voice Conversion
• Joint Training