SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models

2026-06-24

AI Summary

This work addresses the limited ability of existing spoken dialogue models to reason across modalities about paralinguistic social cues, which hinders their capacity to demonstrate authentic emotional intelligence (EI) in multi-turn conversations. To bridge this gap, the study introduces the first EI evaluation framework tailored for multi-turn spoken dialogues, grounded in the EQ-i 2.0 theoretical model. It presents a novel dataset comprising 2,265 multi-turn dialogues and proposes a Spoken EQ (SEQ) scoring protocol inspired by human EI assessment practices. Through multimodal speech-language modeling and paralinguistic feature analysis, the research identifies three critical bottlenecks in current systems: “modality shortcuts,” “safety traps,” and “contextual amnesia.” Experiments reveal that end-to-end spoken language models outperform cascaded architectures yet remain overly reliant on textual content. The dataset and an interactive demo platform are publicly released.

Abstract

As multimodal conversational systems increasingly engage in spoken interaction, their ability to navigate paralinguistic social cues has become a critical bottleneck for natural human-AI communication. However, existing evaluations of machine emotional intelligence assess reasoning exclusively through isolated text or passive acoustic perception, overlooking the complex cross-modal reasoning required for active, multi-turn dialogue. We introduce \textsc{SpeechEQ}, a comprehensive framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs). The framework includes a validated dataset of 2,265 dialogues across 15 Emotional Quotient (EQ) subscales grounded in EQ-i 2.0 theory, along with a multi-turn evaluation protocol measured by our proposed Spoken EQ (SEQ) score inspired by human EQ assessments. Experiments show limitations in how both existing Speech Emotion Recognition and end-to-end Speech-Language Models understand and apply paralinguistic cues through speech. While end-to-end architectures outperform cascaded systems, \textsc{SpeechEQ} reveals that current multimodal models remain bottlenecked by a text-reliant ``modality shortcut,'' an alignment-induced ``safety trap,'' and ``contextual amnesia,'' highlighting the barriers to truly emotionally aware AI. Our benchmark can be accessed at https://huggingface.co/datasets/SpeechEQ/SpeechEQ and demo page at https://binomial14.github.io/speecheq-demo/

Problem

• Emotional Intelligence Quotient
• Speech-Language Models
• Paralinguistic Cues
• Multimodal Dialogue
• Socially Aware AI

Innovation

• SpeechEQ
• Emotional Intelligence Quotient
• Speech-Language Models
• Paralinguistic Cues
• Multimodal Dialogue Evaluation