HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech

2026-06-26

AI Summary

This work addresses the limited emotional expressiveness of large language model (LLM)-driven text-to-speech (TTS) systems during supervised fine-tuning, as well as the information conflict and scale mismatch inherent in existing preference optimization approaches. To overcome these challenges, the authors propose the HPRO framework, which introduces a novel differentiable HD-Emo encoder-decoder to disentangle speech into content and style-preference tokens. HPRO further incorporates a hierarchical progressive reward mechanism that aligns optimization objectives across multiple granularities—frame, word, and sentence levels—effectively bridging the gap between frame-level generation and sentence-level rewards. Experimental results demonstrate that HPRO significantly enhances the emotional expressiveness of synthesized speech while preserving linguistic clarity and semantic integrity.

Abstract

Recently, Large Language Model (LLM)-based Text-to-Speech (TTS) models have achieved remarkable naturalness. However, the standard Supervised Fine-Tuning paradigm often converges to statistically averaged prosody, limiting emotional expressiveness. While preference-driven optimization offers a promising alternative, existing approaches suffer from two structural mismatches: information conflict, where content and emotion in a shared latent space produce conflicting gradients, leading to reward hacking and semantic degradation; and scale gap, where sparse sentence-level rewards struggle to guide dense frame-level generation. To overcome these challenges, we propose HPRO, a hierarchical progressive reward optimization framework. Within HPRO, we introduce the HD-Emo codec as a novel differentiable reward model to resolve the information conflict. It extracts speech into distinct content and style preference tokens, structurally isolating emotional optimization from semantic content. Building upon this structured preference space, HPRO bridges the scale gap by progressively aligning frame-, word- and sentence-level objectives. Experiments demonstrate that HPRO significantly enhances emotional expressiveness, while effectively preserving linguistic intelligibility. The code and audio samples are publicly available at https://xxh333.github.io/hpro-demo/.

Problem

• Emotional Text-to-Speech
• Preference Optimization
• Information Conflict
• Scale Gap
• Prosody Modeling

Innovation

• Hierarchical Reward Optimization
• Preference Extraction
• Emotional Text-to-Speech
• Content-Style Disentanglement
• Multi-granularity Alignment