Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS

2026-08-31

AI Summary

研究通过HybridEmo框架解决多情绪TTS控制问题，采用监督微调初始化并结合组相对策略优化方法，提高情感轨迹正确性和混合强度。

Abstract

Natural-language instructions enable flexible control of synthesized speech, yet emotional TTS systems primarily model a single utterance-level affect, leaving multi-emotion control underexplored. We study two complementary multi-emotion TTS tasks: emotion trajectory, which spans several ordered affective stages, and emotion blending, in which multiple emotions coexist throughout an utterance. These tasks expose a supervision mismatch: supervised fine-tuning (SFT) does not explicitly evaluate emotion features, while single-emotion rewards provide neither structure-aware feedback for trajectory completion nor pair-aware feedback for blending. We introduce HybridEmo, a post-training framework that initializes both tasks with SFT and then aligns the speech-token policy through Group Relative Policy Optimization using a sample-aware hybrid reward. For trajectory samples, segment-aligned consistency combines average and weakest-stage evidence to preserve the correctness and completeness of prescribed stages. For blending samples, a GMM-based reward combines frame-level support from the union of target-emotion anchors in an offline emotion space with an utterance-level weaker-target margin. Both branches share an ASR reward and are routed within a unified policy. On MultiEmo-Test, HybridEmo significantly improves trajectory correctness and blending intensity, without a noticeable degradation in speaker similarity. Human evaluation prefers HybridEmo to CosyVoice 3 and EmoVoice-0.5B, with nearly balanced preferences against Qwen3-TTS.

Problem

• multi-emotion TTS
• emotion trajectory
• emotion blending
• supervised fine-tuning (SFT)
• reward

Innovation

• HybridEmo
• multi-emotion TTS
• Group Relative Policy Optimization
• GMM-based reward