MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models

2025-11-02

AI Summary

Existing speech dialogue models (SDMs) are predominantly evaluated for emotional intelligence in single-turn settings, lacking systematic assessment of their capacity for multi-turn dynamic emotion modeling. To address this gap, we introduce MULTI-Bench—the first benchmark dedicated to evaluating emotional intelligence in multi-turn spoken dialogues—comprising five tasks: emotion recognition, empathetic understanding, contextual reasoning, cross-turn emotion tracking, and adaptive emotional response generation, with 3.2K high-quality samples. We propose a hierarchical evaluation framework that unifies emotion comprehension, inference, and supportive response capabilities within a coherent multi-turn dialogue assessment paradigm, accompanied by an open-source, reproducible evaluation toolkit. Experiments across six state-of-the-art SDMs reveal robust performance on basic emotion recognition but significant deficiencies in cross-turn emotion tracking, implicit emotion inference, and contextually adaptive emotional responding. This work establishes the first comprehensive benchmark for multi-turn emotional intelligence, providing a rigorous new standard for SDM development and evaluation.

Technology Category

• Humans and AI: Emotional Intelligence
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Conversational AI/Dialog Systems

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Spoken Dialogue Models (SDMs) have advanced rapidly, yet their ability to sustain genuinely interactive multi-turn conversations remains underexplored, as most benchmarks focus on single-turn exchanges. We introduce Multi-Bench, the first benchmark explicitly designed to evaluate SDMs in multi-turn interactive dialogue with an emphasis on emotional intelligence. Multi-Bench employs a hierarchical structure with a basic track for emotion understanding and reasoning and an advanced track for emotion support and application. It comprises five carefully designed tasks and about 3.2K samples, ranging from emotion recognition to complex reasoning and interactive dialogue, supported by a reproducible evaluation framework. We evaluate six representative SDMs on eight subsets of Multi-Bench. Results show that while current SDMs achieve good performance on basic understanding tasks, they still have room for improvement in advanced multi-turn interactive dialogue and reasoning-related tasks, particularly in emotion awareness and application.

Problem

• Assessing emotional intelligence in multi-turn spoken dialogues
• Evaluating SDMs' emotion understanding versus application abilities
• Measuring gaps in emotion awareness during interactive conversations

Innovation

• Multi-turn interactive benchmark for emotional intelligence
• Hierarchical structure with basic and advanced tracks
• Reproducible framework with five tasks and 3.2K samples