A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models

2026-07-01

AI Summary

This study investigates the geometric properties of emotion control modules in hybrid speech synthesis systems and their impact on controllability. By employing linear probing and local intrinsic dimensionality (LID) analysis, we compare speech language models (SLMs) and conditional flow matching (CFM) as emotion modulation sites, and evaluate single-point versus joint modulation strategies for synthesizing blended emotions. Our findings reveal, for the first time, that SLMs contain low-dimensional, emotion-specific subspaces decoupled from speaker identity, whereas CFM suffers from entangled emotion and speaker representations, limiting cross-speaker generalization. Although joint modulation enhances emotional intensity, it compromises proportional control and speech quality. The results demonstrate that SLMs significantly outperform CFM in emotion controllability, offering practical guidance for multi-site activation strategies and highlighting the critical role of representational geometry in controllable speech generation.

Abstract

While prior work has explored emotion control in hybrid text-to-speech systems, the geometric properties of these modules, and their implications for steerability, remain poorly understood. We present the first comparative study of speech language model (SLM) and conditional flow-matching (CFM) modules as activation steering sites for mixed emotion speech synthesis. We first characterize emotion representations using linear probing and local intrinsic dimensionality (LID), and then evaluate single-site and joint steering for mixed-emotion synthesis. Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement. Joint steering increases emotion intensity but degrades proportional control and speech quality on in-distribution data. These findings provide practical guidance for multi-site activation steering in hybrid TTS systems and highlight the importance of representation geometry in controllable speech generation.

Problem

• emotion steering
• text-to-speech
• representation geometry
• mixed-emotion synthesis
• activation steering

Innovation

• emotion steering
• representation geometry
• speech language model
• conditional flow-matching
• disentanglement