Speaker-agnostic Emotion Vector for Cross-speaker Emotion Intensity Control

2025-07-04

AI Summary

Cross-speaker emotional intensity control suffers from speaker identity degradation, primarily due to incompatibility between source and target speakers’ emotion embeddings. To address this, we propose a speaker-agnostic emotion vector modeling framework: emotion features are disentangled across multiple speakers, and a shared emotion space is constructed to yield generalizable, speaker-invariant emotion representations. Building upon this, we introduce an emotion arithmetic paradigm that enables arbitrary-intensity emotional speech synthesis using only the target speaker’s neutral utterances—no emotional speech data from the target speaker is required. The method supports both seen and unseen speakers, significantly improving cross-speaker emotional controllability, speech naturalness, and speaker fidelity. Experimental results demonstrate superior performance over state-of-the-art approaches in emotion intensity accuracy, MOS scores, and speaker similarity.

Technology Category

• Humans and AI: Emotional Intelligence
• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Cross-speaker emotion intensity control aims to generate emotional speech of a target speaker with desired emotion intensities using only their neutral speech. A recently proposed method, emotion arithmetic, achieves emotion intensity control using a single-speaker emotion vector. Although this prior method has shown promising results in the same-speaker setting, it lost speaker consistency in the cross-speaker setting due to mismatches between the emotion vector of the source and target speakers. To overcome this limitation, we propose a speaker-agnostic emotion vector designed to capture shared emotional expressions across multiple speakers. This speaker-agnostic emotion vector is applicable to arbitrary speakers. Experimental results demonstrate that the proposed method succeeds in cross-speaker emotion intensity control while maintaining speaker consistency, speech quality, and controllability, even in the unseen speaker case.

Problem

• Achieve cross-speaker emotion intensity control using neutral speech
• Overcome speaker inconsistency in emotion vector mismatch
• Develop speaker-agnostic emotion vector for arbitrary speakers

Innovation

• Speaker-agnostic emotion vector design
• Cross-speaker emotion intensity control
• Maintaining speaker consistency and quality