Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder

2026-08-12

AI Summary

This work addresses the limitation of existing zero-shot text-to-speech (TTS) systems in cross-lingual voice cloning, which typically rely on transcriptions of reference audio and thus struggle in real-world scenarios where such transcripts are unavailable. To overcome this, the authors propose a transcription-free multilingual zero-shot TTS system featuring a two-stage architecture: a large language model–based text-to-semantic generation module followed by a conditional flow matching–based semantic-to-acoustic synthesis module. A learnable speaker encoder is introduced to extract prosodic and timbral characteristics directly from self-supervised speech representations, enabling both intra- and cross-lingual voice cloning across 14 languages. Experimental results on the CV3-Eval benchmark show an average word error rate of 3.73%, and human evaluations indicate superior overall performance compared to current open-source and commercial systems.

Abstract

Recent advances in zero-shot text-to-speech (TTS) have substantially improved speech quality and voice cloning fidelity. However, many zero-shot TTS systems still depend on audio prompt transcripts at inference time. This dependency limits cross-lingual voice cloning, since in-the-wild reference audio is often untranscribed. In this technical report, we present Confucius4-TTS, a multilingual zero-shot TTS system that supports 14 languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio prompts. Confucius4-TTS follows a two-stage architecture, consisting of text-to-semantic (T2S) and semantic-to-acoustic (S2A) modules. The LLM-based T2S module uses a learnable speaker encoder to extract timbre features from self-supervised speech representations, and the conditional flow-matching S2A module converts the predicted semantic tokens into mel-spectrograms. The same model also supports continuation cloning when a reference transcript is available. Confucius4-TTS is trained on large-scale multilingual speech data. It achieves high intelligibility and speaker similarity on public benchmarks. On the CV3-Eval cross-lingual benchmark, Confucius4-TTS obtains an average WER of 3.73% across six directions. On our internal cross-lingual set, it achieves the best average overall rank in human evaluation among recent open-source and commercial systems. We release code, model checkpoints, and demos at https://github.com/netease-youdao/Confucius4-TTS.

Problem

• zero-shot TTS
• cross-lingual voice cloning
• transcript-free
• audio prompt
• multilingual speech

Innovation

• transcript-free
• cross-lingual zero-shot TTS
• learnable speaker encoder
• self-supervised speech representation
• conditional flow-matching