Multimodal Emotion Recognition with Large Language Models

2026-05-20

AI Summary

This study addresses key challenges in multimodal emotion recognition—namely, the scarcity of affectively annotated data, semantic gaps across modalities, and opaque reasoning processes—by proposing a novel paradigm termed “MER-with-LLMs.” The work systematically constructs a large language model (LLM)-based framework that integrates three core components: affective data augmentation, cross-modal alignment, and interpretable reasoning. It establishes, for the first time, a comprehensive taxonomy and research roadmap for leveraging LLMs in general-purpose affective intelligence. Through a thorough survey of existing approaches, the paper delineates a clear academic landscape, elucidating the field’s developmental trajectory, critical bottlenecks, and emerging directions, thereby advancing multimodal emotion recognition toward greater structural coherence and interpretability.

Abstract

Multimodal Emotion Recognition (MER) focuses on identifying and interpreting emotions from modality-compound inputs. Closely mirroring human cognitive processes in real-world environments, MER has drawn substantial attention from both academia and industry. Recently, a paradigm shift has been unveiled in MER, from leveraging small-scale, task-specific models to Large Language Models (LLMs). We refer to the latter as the MER-with-LLMs paradigm, which offers unprecedented generality, spurring numerous empirical attempts, even alongside speculation about LLMs' potential to achieve general emotional intelligence. However, with these new opportunities come new challenges, including the scarcity of emotionally annotated data, the affective gap both within and across modalities, and the opacity of affective interpretation. To systematically review existing research and guide future exploration, this paper categorizes prior works according to their focus on addressing these challenges into three directions: Affective Data Augmentation, Multimodal Affective Representation, and Multimodal Affective Reasoning. By thoroughly tracing the development, emerging trends, and remaining issues within each direction, this paper aims to provide a clear academic map of the MER-with-LLMs paradigm and foster its structured advancement.

Problem

• Multimodal Emotion Recognition
• Large Language Models
• Affective Gap
• Emotion Annotation
• Interpretability

Innovation

• Multimodal Emotion Recognition
• Large Language Models
• Affective Data Augmentation
• Multimodal Affective Representation
• Multimodal Affective Reasoning