Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis

2025-12-21

AI Summary

To address the challenge of cross-style text-to-speech (TTS) stemming from scarce jointly labeled dialect–emotion data, this paper proposes Hierarchical Expressive Vectors (HE-Vectors), enabling zero-shot emotional dialect speech synthesis for the first time. Our method adopts the task vector paradigm and achieves dialect–emotion style disentanglement via decoupled modeling: a weighted expressive vector (E-Vector) captures style-specific attributes, while a hierarchical vector fusion mechanism supports independent control and joint generation of both dimensions—eliminating reliance on jointly annotated data entirely. Under zero-shot conditions—i.e., without any dialect–emotion paired labels—the synthesized speech achieves a Mean Opinion Score (MOS) of 4.12, significantly outperforming existing baselines. This demonstrates the successful unification of style disentanglement, fine-grained controllability, and speech naturalness.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Emotional Intelligence

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advances in text-to-speech (TTS) have yielded remarkable improvements in naturalness and intelligibility. Building on these achievements, research has increasingly shifted toward enhancing the expressiveness of generated speech, such as dialectal and emotional TTS. However, cross-style synthesis combining both dialect and emotion remains challenging and largely unexplored, mainly due to the scarcity of dialectal data with emotional labels. To address this, we propose Hierarchical Expressive Vector (HE-Vector), a two-stage method for Emotional Dialectal TTS. In the first stage, we construct different task vectors to model dialectal and emotional styles independently, and then enhance single-style synthesis by adjusting their weights, a method we refer to as Expressive Vector (E-Vector). For the second stage, we hierarchically integrate these vectors to achieve controllable emotionally expressive dialect synthesis without requiring jointly labeled data, corresponding to Hierarchical Expressive Vector (HE-Vector). Experimental results demonstrate that HE-Vectors achieve superior performance in dialect synthesis, and promising results in synthesizing emotionally expressive dialectal speech in a zero-shot setting.

Problem

• Addresses cross-style synthesis of dialect and emotion in TTS
• Overcomes scarcity of labeled dialectal data with emotions
• Enables controllable expressive dialect synthesis without joint data

Innovation

• Two-stage hierarchical vector integration method
• Independent modeling of dialect and emotion styles
• Zero-shot synthesis without jointly labeled data