Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions

2025-06-03

AI Summary

Existing emotional text-to-speech (TTS) systems are largely constrained to predefined discrete emotion categories, limiting their ability to model continuous, mixed, and fine-grained emotional expressions prevalent in natural dialogue. To address this, we propose the first prompt-driven zero-shot unseen emotion modeling framework for TTS. Our method leverages large language models (LLMs) to guide context-aware prompt learning, enabling controllable synthesis of arbitrary novel emotions—including blended affective states—without requiring labeled emotional speech data. Technically, it integrates dynamic emotion weight quantization with context-aware knowledge distillation and embeds a lightweight prompt adaptation module into the TTS backbone. Experiments demonstrate that our approach generates high-fidelity, emotionally consistent, and diverse speech in zero-shot settings, significantly outperforming conventional classification-based TTS systems. This work establishes a new paradigm for natural, adaptive human–machine affective interaction.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Existing expressive text-to-speech (TTS) systems primarily model a limited set of categorical emotions, whereas human conversations extend far beyond these predefined emotions, making it essential to explore more diverse emotional speech generation for more natural interactions. To bridge this gap, this paper proposes a novel prompt-unseen-emotion (PUE) approach to generate unseen emotional speech via emotion-guided prompt learning. PUE is trained utilizing an LLM-TTS architecture to ensure emotional consistency between categorical emotion-relevant prompts and emotional speech, allowing the model to quantitatively capture different emotion weightings per utterance. During inference, mixed emotional speech can be generated by flexibly adjusting emotion proportions and leveraging LLM contextual knowledge, enabling the model to quantify different emotional styles. Our proposed PUE successfully facilitates expressive speech synthesis of unseen emotions in a zero-shot setting.

Problem

• Generating expressive speech for unseen mixed emotions
• Overcoming limitations of predefined categorical emotions in TTS
• Quantifying emotional styles via LLM contextual knowledge

Innovation

• Emotion-guided prompt learning for unseen emotions
• LLM-TTS architecture for emotional consistency
• Flexible emotion proportion adjustment in synthesis