EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment

2025-04-23

AI Summary

Existing multimodal large language models (MLLMs) lack systematic evaluation of image-evoked emotion understanding, with prevailing benchmarks suffering from coarse granularity and narrow task scope. Method: We introduce EmoMLLM—the first dedicated benchmark for image emotion understanding—comprising four task categories: perception, fine-grained ranking, descriptive generation, and holistic assessment, augmented by pairwise image comparison. We propose a VAD (Valence-Arousal-Dominance)-based fine-grained emotion ranking annotation paradigm, supported by 1,960 human-annotated images and 6,773 QA pairs. Additionally, we design single- and dual-sample joint analysis tasks to enable multidimensional capability evaluation. Results: Empirical evaluation across 19 state-of-the-art MLLMs reveals significant deficiencies in fine-grained emotion discrimination and comparative reasoning. EmoMLLM establishes a reproducible, extensible evaluation infrastructure for machine empathy modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The furnishing of multi-modal large language models (MLLMs) has led to the emergence of numerous benchmark studies, particularly those evaluating their perception and understanding capabilities. Among these, understanding image-evoked emotions aims to enhance MLLMs' empathy, with significant applications such as human-machine interaction and advertising recommendations. However, current evaluations of this MLLM capability remain coarse-grained, and a systematic and comprehensive assessment is still lacking. To this end, we introduce EEmo-Bench, a novel benchmark dedicated to the analysis of the evoked emotions in images across diverse content categories. Our core contributions include: 1) Regarding the diversity of the evoked emotions, we adopt an emotion ranking strategy and employ the Valence-Arousal-Dominance (VAD) as emotional attributes for emotional assessment. In line with this methodology, 1,960 images are collected and manually annotated. 2) We design four tasks to evaluate MLLMs' ability to capture the evoked emotions by single images and their associated attributes: Perception, Ranking, Description, and Assessment. Additionally, image-pairwise analysis is introduced to investigate the model's proficiency in performing joint and comparative analysis. In total, we collect 6,773 question-answer pairs and perform a thorough assessment on 19 commonly-used MLLMs. The results indicate that while some proprietary and large-scale open-source MLLMs achieve promising overall performance, the analytical capabilities in certain evaluation dimensions remain suboptimal. Our EEmo-Bench paves the path for further research aimed at enhancing the comprehensive perceiving and understanding capabilities of MLLMs concerning image-evoked emotions, which is crucial for machine-centric emotion perception and understanding.

Problem

• Assessing image-evoked emotions in multi-modal large language models
• Lacking systematic evaluation for emotion understanding in MLLMs
• Developing a benchmark to enhance MLLMs' emotion perception capabilities

Innovation

• Emotion ranking strategy with VAD attributes
• Four tasks for MLLM emotion evaluation
• Image-pairwise analysis for joint comparison