Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale

2025-11-26

AI Summary

Single-codebook text-to-speech (TTS) large language models suffer from critical issues including prosodic instability, speaker drift, and degraded naturalness. To address these, we propose a multi-reward GRPO (Generalized Reinforcement Learning with Policy Optimization) framework that optimizes discrete speech token generation via reinforcement learning. Our method introduces length penalties and entropy regularization to stabilize sequence modeling; designs an LLM-driven prosody alignment reward; and leverages an external reasoning LLM to generate diverse pause structures—yielding high-quality human preference signals. Additionally, we integrate flow-matching decoding to further enhance output quality. Experiments demonstrate consistent and significant improvements across prosodic consistency, speaker similarity, and MOS naturalness, under varying data scales and model sizes. Moreover, synergistic integration with flow-matching decoders yields additional gains, validating the framework’s generality and effectiveness.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent advances in Large Language Models (LLMs) have transformed text-to-speech (TTS) synthesis, inspiring autoregressive frameworks that represent speech as sequences of discrete codec tokens. Among them, single-codebook TTS LLMs have emerged as compact and streamable architectures that jointly model semantic and acoustic integration. However, despite their efficiency, these models often exhibit unstable prosody, speaker drift, and degraded naturalness. To address these issues, we propose a multi-reward Group Relative Policy Optimization (GRPO) framework that directly optimizes the token generation policy of single-codebook TTS LLMs. Beyond standard intelligibility and speaker similarity objectives, our design integrates three rule-based rewards: a length penalty for duration consistency, an entropy regularization reward for decoding stability, and an LLM-annotated prosody alignment reward that explicitly supervises rhythm. In this prosody reward, an external reasoning LLM predicts multiple plausible pause structures via in-context learning, providing a human-preference-aligned supervisory signal for GRPO training. To assess universality, we further attach a flow-matching (FM) decoder on top of the GRPO-optimized AR backbone and observe consistent additional gains, indicating that our reinforcement optimization enhances the intrinsic AR policy. We further conduct a scalability analysis across data sizes and model scales, revealing that the proposed method consistently enhances prosodic stability, speaker similarity, and overall speech naturalness in single-codebook TTS LLMs.

Problem

• Single-codebook TTS LLMs suffer from unstable prosody and speaker drift
• These models exhibit degraded speech naturalness despite their efficiency
• The research addresses inconsistent duration and decoding stability issues

Innovation

• Multi-reward GRPO optimizes TTS token generation policy
• Integrates length penalty, entropy regularization, prosody alignment rewards
• Uses external LLM for pause structure prediction in training