Enhancing Flow Matching with A Unified Guidance Framework for Efficient and Robust Speech Synthesis

2026-06-30

AI Summary

Flow Matching in speech synthesis suffers from high inference latency and timbre leakage. This work proposes a unified guidance framework that, for the first time, jointly integrates data-level and model-level guidance. By leveraging heterogeneous data augmentation to disentangle linguistic content from acoustic residuals, and combining trajectory correction with an intrinsic guidance objective, the method distills conditional information directly into network weights to optimize the inference trajectory. Notably, it eliminates the need for Classifier-Free Guidance, substantially reducing computational overhead. The approach achieves nearly threefold faster inference while preserving high timbre fidelity and significantly outperforms state-of-the-art baselines in speaker similarity.

Abstract

Flow Matching (FM) has emerged as a powerful paradigm for speech generation but remains constrained by high inference latency and timbre leakage. To address these bottlenecks, we propose a unified guidance framework that enhances generation efficiency and robustness through two complementary strategies. On the data front, we introduce Data-guidance via heterogeneous augmentation, encouraging the model to disentangle linguistic content from acoustic residue. In parallel, we propose an enhanced Model-guidance mechanism that synergizes trajectory rectification with a novel intrinsic guidance objective. This approach distills conditional knowledge into network weights and straightens inference trajectory path, thereby eliminating Classifier-Free Guidance (CFG) overhead. Experiments demonstrate that our framework accelerates inference by nearly three times while effectively improving speaker similarity compared to state-of-the-art baselines.

Problem

• Flow Matching
• inference latency
• timbre leakage
• speech synthesis
• robustness

Innovation

• Flow Matching
• Unified Guidance Framework
• Data-guidance
• Model-guidance
• Classifier-Free Guidance