How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech

2026-06-18

AI Summary

This study addresses the limited understanding of how natural language instructions modulate acoustic outputs in current stylized text-to-speech (TTS) systems, which hinders model controllability and failure attribution. For the first time, the diffusion attention attribution method (DAAM) is introduced to the speech generation domain to perform cross-attention attribution analysis across 25 network layers and 24 ODE steps of the CapSpeech-TTS model, enabling fine-grained visualization and quantification of the influence of style-descriptive words. The findings reveal that style words exert a global regulatory effect, with their attention intensity significantly correlated with fundamental frequency and energy. This influence is most pronounced in deeper network layers—particularly layer 17, which exhibits the strongest selectivity—and during early ODE integration steps.

Abstract

Style-captioned text-to-speech systems use natural language to control voice characteristics, but how individual words influence acoustic output remains unclear. Understanding this is critical for diagnosing failure modes and improving controllability in expressive TTS. We propose cross-attention attribution for speech diffusion models, adapting the DAAM framework to the speech domain for the first time, and apply it to CapSpeech-TTS. Our method extracts per-token heatmaps across 25 layers and 24 ODE steps. We analyze 3,600 (style caption, text transcript) combinations comprising 120 style captions conditioning the generation of 30 text transcripts each, revealing how caption tokens shape waveforms. Results show: (1) style tokens have lower temporal variance than content/function tokens, confirming global conditioning; (2) style attention correlates with F0 and energy; (3) style conditioning peaks in early steps and deep layers; (4) attention entropy reaches its minimum at layer 17, co-occurring with the style importance peak, indicating maximal network selectivity at the most style-critical stage. This is the first study of how natural language influences cross-attention in speech diffusion models

Problem

• style-captioned TTS
• cross-attention
• speech diffusion models
• natural language control
• acoustic output

Innovation

• cross-attention attribution
• speech diffusion models
• style-captioned TTS
• DAAM adaptation
• attention entropy