Fréchet Distance Loss on Speech Representations for Text-to-Speech Synthesis

2026-07-07

AI Summary

This work addresses the challenge that few-step diffusion and flow-matching text-to-speech (TTS) models, due to their reliance on local training objectives, often fail to align the distribution of synthesized speech with that of high-quality reference speech, thereby compromising intelligibility. To mitigate this, the authors propose incorporating a Speech Representation Fréchet Distance (SR-FD) loss during fine-tuning, which—without requiring a discriminator or incurring inference overhead—acts as a distributional regularizer by matching the mean and covariance of synthesized and reference speech in a semantic representation space. This space is constructed using a frozen Whisper encoder and a CTC-based feature extractor. Evaluated on the Seed-TTS English dataset, four-step fine-tuning with SR-FD reduces the word error rate from 2.2279% to 1.4147% (a relative improvement of 36.5%) compared to the original ten-step baseline, while preserving comparable audio quality and speaker similarity.

Abstract

Few-step diffusion and flow-matching text-to-speech (TTS) models are usually trained with local objectives, such as conditional flow matching, reconstruction, and stop prediction. These losses provide stable optimization, but they never ask whether sampled speech follows the distribution of high-quality speech. We propose Speech Representation Fr'echet Distance loss (SR-FD), a training-time distributional regularizer for tokenizer-free flow-matching autoregressive TTS. During fine-tuning, the model synthesizes speech with the same few-step sampler used at deployment, and SR-FD matches the mean and covariance of frozen Whisper and CTC features of this speech to reference statistics computed offline from three complementary content targets. The loss requires no discriminator and no inference-time computation. On Seed-TTS English, four-step SR-FD fine-tuning reduces WER from the original four-step VoxCPM2 baseline's 2.2279% to 1.4147%, a 36.5% relative reduction, and also surpasses the original ten-step baseline at 1.7366%; both gains are significant under an utterance-level paired bootstrap. Speaker similarity and objective quality proxies are preserved at the ten-step level, and an error analysis shows the gain comes from content substitutions across all prompt lengths. SR-FD is thus an intelligibility-improving distributional regularizer for few-step TTS.

Problem

• Text-to-Speech Synthesis
• Few-step TTS
• Speech Distribution
• Intelligibility
• Fréchet Distance

Innovation

• Fréchet Distance
• distributional regularization
• few-step TTS
• flow matching
• speech intelligibility