AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence

2026-05-20

AI Summary

Current evaluations of emotional intelligence in large language models often rely on synthetic data or single-turn interactions, failing to capture their capacity to perceive and respond to user emotions in authentic multi-turn dialogues. This work introduces a new benchmark based on 200 real human–AI multi-turn conversations, featuring fine-grained annotations of users’ emotional states, model behaviors, and desired responses at each turn. For the first time, emotional intelligence is decomposed into distinct, independently assessable sub-competencies. Experiments across 11 models reveal that these sub-competencies are largely independent, with preference alignment and response quality proving more discriminative than emotion recognition accuracy, thereby underscoring the critical role of context-sensitive responsiveness in emotionally intelligent interaction.

Abstract

Emotional intelligence (EI), the ability to perceive, understand, and respond appropriately to others' emotional states, is central to human communication, and increasingly important to assess as LLMs assume conversational roles in everyday life. Existing EI benchmarks rely on synthetic prompts, single-turn cases, or third-party annotation. These approaches do not directly measure how models infer and respond to a participant's emotional state over the course of a real conversation. We introduce AttuneBench, a benchmark grounded in 200 genuine multi-turn human-model conversations in which participants conversed with anonymized LLMs and provided turn-by-turn annotations of their emotional state, the model's behavior, and their preferred responses. Across 11 evaluated models, we find that model rankings on emotion recognition, behavioral classification, preference prediction, and judged response quality are largely independent, indicating that emotionally intelligent behavior decomposes into separable capabilities. Preference alignment and response-quality judgments are substantially more model-discriminating than emotion-label accuracy. These results indicate that emotionally intelligent behavior requires predicting what kind of response a specific user wants in context, a distinction that aggregate scoring can obscure and that single-turn or synthetic formats cannot directly capture across turns. AttuneBench provides a framework for assessing each of these capabilities and for diagnosing model-specific strengths and failure modes in emotionally salient conversation.

Problem

• emotional intelligence
• large language models
• conversation-based benchmark
• emotion recognition
• preference alignment

Innovation

• Emotional Intelligence
• Multi-turn Conversation
• Preference Alignment
• Response Quality
• Human-Model Interaction