EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness

2026-05-01

AI Summary

This work addresses the susceptibility of multimodal large language models to emotional recognition bias under conditions of modality conflict or absence, particularly manifesting as a significant attenuation of video modality contribution—termed “Video Contribution Collapse” (VCC). To systematically evaluate model behavior across aligned, conflicting, and missing modality scenarios, the authors introduce the EmoMM benchmark. They further propose CHASE, a training-free, inference-time conflict-aware attention steering mechanism that dynamically modulates cross-modal attention to mitigate such biases. Experimental results demonstrate that CHASE consistently enhances emotional recognition performance across diverse settings, substantially improving model robustness and accuracy in complex, real-world multimodal contexts.

Abstract

Multimodal Emotion Recognition (MER) is critical for interpreting real-world interactions. While Multimodal Large Language Models (MLLM) have shown promise in MER, their internal decision-making mechanisms under modality conflict and missingness remain largely underexplored. In this paper, to systematically investigate these behaviors, we introduce EmoMM, a comprehensive benchmark featuring modality-aligned, conflict, and missing subsets. Through extensive evaluation, we uncover a Video Contribution Collapse (VCC) phenomenon, where MLLM marginalize video evidence due to high token redundancy and modality preferences. To address this, we propose Conflict-aware Head-level Attention Steering (CHASE), a lightweight mechanism that detects modality conflicts and performs inference-time attention steering, effectively mitigating decision bias without retraining the backbone. Experimental results demonstrate that CHASE consistently improves performance across various settings, significantly enhancing the reliability of MLLM in complex affective scenarios.

Problem

• Multimodal Emotion Recognition
• Modality Conflict
• Modality Missingness
• Video Contribution Collapse
• MLLM Decision Bias

Innovation

• Multimodal Emotion Recognition
• Modality Conflict
• Attention Steering
• Video Contribution Collapse
• Inference-time Intervention