AffectVerse: Emotional World Models for Multimodal Affective Computing

2026-05-19

AI Summary

This study addresses the limitation of existing emotion recognition approaches that treat emotion as a static multimodal fusion, thereby neglecting its inherently dynamic nature. To explicitly model the temporal evolution of emotions, this work proposes an Emotion World Module (EWM) built upon Qwen2.5-Omni, which incorporates action-free, short-term emotion prediction at the representation level. The framework integrates cross-modal temporal imagination, modality-aware multi-step attention (MAMA) for belief aggregation, and belief injection to capture evolving emotional states. Notably, it leverages future prediction as a self-supervised signal, enabling emotion belief state modeling without requiring additional inputs during inference. Evaluated across nine benchmark datasets, the method achieves an average performance gain of at least 2.57%, with ablation studies confirming the additive contributions of each component.

Abstract

Humans infer emotions by integrating observed multimodal cues with expectations about how affective states may unfold. Existing multimodal large language models (MLLMs), however, often treat emotion recognition as static fusion over complete audiovisual-text inputs, leaving affective dynamics implicit. We propose AffectVerse, a Qwen2.5-Omni-based model equipped with an Emotion World Module (EWM), an action-free representation-level module for short-horizon latent affective prediction. \rev{EWM contains three modules: 1) Cross-Modal Temporal Imagination predicts future video/audio representations from past tokens with multi-step rollout. 2) MAMA(Modality-Aware Multi-step Attention) Belief Aggregation compresses imagined tokens into modality-aware belief tokens. 3) Belief Injection inserts these belief tokens into the LLM for affective reasoning.} AffectVerse uses future prediction as a past-conditioned self-supervised signal: it does not replace modeling observed history or require unseen signals at inference, but forces the current belief state to encode transition cues that are predictive of subsequent affective change. Across nine benchmarks, AffectVerse improves at least 2.57\% over other models, while controlled ablations show additive gains from temporal imagination, cross-modal rollout, and belief aggregation. These results suggest predictive belief-state modeling is a practical alternative for affective computing.

Problem

• affective computing
• multimodal emotion recognition
• emotional dynamics
• temporal modeling
• world models

Innovation

• Emotion World Module
• Temporal Imagination
• Belief Aggregation
• Multimodal Affective Computing
• Predictive World Models