Cross-Lingual F5-TTS 2: A Simplified Framework for Language-Agnostic Voice Cloning

2026-09-14

AI Summary

本文提出Cross-Lingual F5-TTS 2，通过预训练模型和细调解决无文本跨语言声音克隆问题，避免了强制对齐的复杂性和误差。

Abstract

Zero-shot text-to-speech (TTS) can clone a speaker's voice from a short audio prompt, yet most TTS systems still require the audio prompt transcript during inference. This dependency prevents cross-lingual voice cloning when the audio prompt transcript is unavailable, particularly for unseen languages. Cross-Lingual F5-TTS removes this dependency and enables transcript-free cross-lingual voice cloning, but it prepares its training data with forced alignment. Forced alignment is sensitive to boundary errors, and its cost grows as more languages are covered. Its speaking rate predictor is also unreliable at estimating duration when the audio prompt begins or ends with silence. In this paper, we present Cross-Lingual F5-TTS 2, a simplified framework for transcript-free cross-lingual voice cloning without forced alignment. Instead of using forced alignment to segment real utterances, we build same-speaker prompt and target pairs using a pretrained F5-TTS model and fine-tune the same model on these constructed pairs. This simplifies data preparation and preserves the acoustic modeling capability of the pretrained model, enabling adaptation with only a short fine-tuning stage. We further make the syllable-level speaking rate predictor robust to leading and trailing silence through silence-aware augmentation. Experiments show that Cross-Lingual F5-TTS 2 reaches higher speaker similarity than F5-TTS and Cross-Lingual F5-TTS while maintaining intelligibility. All related resources are publicly available.

Problem

• Cross-Lingual Voice Cloning
• Zero-shot TTS
• Forced Alignment
• Transcript-Free

Innovation

• transcript-free
• cross-lingual voice cloning
• forced alignment
• silence-aware augmentation