RE-LLM: Refining Empathetic Speech-LLM Responses by Integrating Emotion Nuance

2026-02-11

AI Summary

This work addresses the limited emotional depth in empathetic responses generated by current large language models, which struggle to capture nuanced affective cues from text alone. To overcome this limitation, the study proposes a novel speech-enabled large language model that integrates dimensional emotion representations with vocal information for the first time. The framework incorporates auxiliary learning and multi-task training mechanisms to enhance both emotional understanding and expressive capability in empathetic dialogue. Experimental results on the ESD, IEMOCAP, and MSP-PODCAST datasets demonstrate significant performance improvements: the proposed approach achieves up to a 14.79% increase in emotional response scores, a remarkable 139.28% gain in emotional exploration scores, and a 6.9% improvement in speech-based emotion recognition accuracy.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Emotional Intelligence

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

With generative AI advancing, empathy in human-AI interaction is essential. While prior work focuses on emotional reflection, emotional exploration, key to deeper engagement, remains overlooked. Existing LLMs rely on text which captures limited emotion nuances. To address this, we propose RE-LLM, a speech-LLM integrating dimensional emotion embeddings and auxiliary learning. Experiments show statistically significant gains in empathy metrics across three datasets. RE-LLM relatively improves the Emotional Reaction score by 14.79% and 6.76% compared to text-only and speech-LLM baselines on ESD. Notably, it raises the Exploration score by 35.42% and 3.91% on IEMOCAP, 139.28% and 9.83% on ESD, and 60.95% and 22.64% on MSP-PODCAST. It also boosts unweighted accuracy by 5.4% on IEMOCAP, 2.3% on ESD, and 6.9% on MSP-PODCAST in speech emotion recognition. These results highlight the enriched emotional understanding and improved empathetic response generation of RE-LLM.

Problem

• empathy
• emotion nuance
• speech-LLM
• emotional exploration
• human-AI interaction

Innovation

• speech-LLM
• dimensional emotion embeddings
• empathetic response generation
• auxiliary learning
• emotion exploration