Rationale-Guided Learning for Multimodal Emotion Recognition

2026-08-11

AI Summary

This work addresses the limitations of existing multimodal conversational emotion recognition approaches, which typically rely on end-to-end mappings and fail to model the causal reasoning processes underlying human emotion understanding. Inspired by dual-process cognitive theory, the authors decompose emotion recognition into three stages—intuitive, contextual, and integrative reasoning—and introduce, for the first time, a cognition-inspired structured reasoning mechanism. By leveraging large language models to generate structured rationales offline as memory-guided priors, they train a multimodal large language model to achieve representation alignment, enabling it to emulate human-like emotion comprehension during inference without additional computational overhead. The proposed method achieves state-of-the-art performance on IEMOCAP and MELD benchmarks and demonstrates strong generalization by retrieving semantically consistent reasoning rationales for unseen samples, offering both high accuracy and interpretability.

Abstract

Multimodal emotion recognition in conversation (MERC) requires understanding complex interactions between verbal and non-verbal cues. However, most existing approaches fundamentally treat this as a direct input-output (multimodal cues-emotion labels) mapping problem, overlooking the causal reasoning that humans use when interpreting emotions. We propose rationale-guided learning (RGL), a novel framework that transforms MERC into a cognitively-inspired reasoning task. Based on dual-process theory, we decompose emotional reasoning into three facets: Intuitive (immediate perception, System 1), Contextual (situational analysis, System 2), and Integrative (synthesis of both). We leverage an MLLM offline to generate structured rationales, which are encoded as memories to guide model training via aligning internal representations with human-like reasoning patterns. Our final model operates without any MLLM overheads at inference time. Experimental results show that RGL achieves state-of-the-art performance on the IEMOCAP and MELD benchmarks. Further, for interpretation, we demonstrate that the model's internal features effectively retrieve semantically correct rationales for unseen test samples, validating its rationale reasoning capabilities.

Problem

• Multimodal Emotion Recognition
• Emotion Recognition in Conversation
• Causal Reasoning
• Human-like Reasoning
• Rationale

Innovation

• Rationale-Guided Learning
• Multimodal Emotion Recognition
• Dual-Process Theory
• Cognitive Reasoning
• Interpretable AI