FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech

2026-06-22

AI Summary

This work addresses the limited exploration of flow-matching methods in reinforcement learning (RL) for text-to-speech (TTS) synthesis, where existing approaches predominantly focus on large language models. The paper proposes FlowTTS-GRPO, the first framework to apply online RL directly to end-to-end fine-tuning of open-source flow-matching TTS models. It reformulates the ordinary differential equation (ODE) trajectory as a stochastic differential equation (SDE) path and introduces a weighted multi-objective reward mechanism to replace conventional probabilistic guidance. Key findings reveal that omitting classifier-free guidance (CFG), emphasizing hard examples, and applying RL exclusively to the flow-matching component significantly enhance performance. Evaluated on CosyVoice 3.0 and F5-TTS, the method consistently improves speaker similarity and perceptual audio quality, with F5-TTS additionally achieving notable gains in intelligibility, outperforming baselines across both subjective and objective metrics.

Abstract

Existing Reinforcement Learning (RL) research for Text-to-Speech (TTS) focuses on large language models (LLMs), leaving Flow-Matching (FM) under-explored. We present FlowTTS-GRPO, an online RL framework for FM-based TTS. By converting ordinary differential equation (ODE) trajectories into stochastic differential equation (SDE) paths, our method enables direct fine-tuning of open-source FM models without auxiliary models. We show that a weighted reward combination converges faster than a probabilistic scheme, and identify three practical optimizations: omitting classifier-free guidance (CFG) during training accelerates convergence; synthesizing hard cases improves robustness; and applying RL to the FM component enhances audio-detail metrics. Experiments on CosyVoice 3.0 and F5-TTS demonstrate objective and subjective preference gains in speaker similarity and perceptual quality, with F5-TTS also improving intelligibility.

Problem

• Text-to-Speech
• Flow-Matching
• Reinforcement Learning
• Multi-Objective Reward
• Online Learning

Innovation

• Flow-Matching
• Reinforcement Learning
• Text-to-Speech
• Online Fine-tuning
• Multi-Objective Reward