Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTS

2026-06-24

AI Summary

This work addresses the challenge in flow-matching-based zero-shot text-to-speech synthesis where conventional classifier-free guidance struggles to simultaneously preserve textual accuracy and speaker similarity. To resolve this, the authors propose a conditionally decoupled guidance approach that independently masks text and speech prompts, decomposing the guidance field into distinct text, speaker, and joint residual components. A joint residual reweighting mechanism is introduced to enable independent control over speaker characteristics and residual terms. This method effectively decouples speaker representation from the text generation process within the standard classifier-free guidance framework, thereby preventing mutual interference. Experimental results on F5-TTS and CosyVoice2 demonstrate that the proposed approach significantly enhances speaker similarity while maintaining high text correctness.

Abstract

Classifier-free guidance (CFG) is widely used in flow-matching-based zero-shot text-to-speech (TTS), where generation is typically controlled by two conditions: the target text and a prompt speech signal. Standard CFG strengthens these conditions jointly, while recent branch-selective guidance methods attempt to enhance text or speaker conditioning separately, often leading to a trade-off between text correctness and speaker similarity. In this paper, we revisit the CFG under independently masked text and speech-prompt conditions, and decompose the guidance field into text, speaker, and joint residuals. We show that conventional speaker-selective guidance entangles the speaker residual with the joint residual, which may disturb text-related generation. Based on this observation, we propose joint residual reweighting, which independently controls the speaker and joint residuals within the standard CFG framework. Experiments on F5-TTS and CosyVoice2 show that the proposed method improves speaker similarity while maintaining competitive text correctness, demonstrating the usefulness of the joint residual for balancing speaker fidelity and text accuracy in zero-shot TTS.

Problem

• classifier-free guidance
• zero-shot TTS
• flow matching
• speaker similarity
• text correctness

Innovation

• classifier-free guidance
• flow-matching
• zero-shot TTS
• residual decomposition
• joint residual reweighting