Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

2026-06-23

AI Summary

This work addresses the limitations of existing emotion-oriented general-purpose multimodal large language models, which often fail to fully exploit multimodal cues and suffer from cross-modal hallucinations, leading to unreliable perception. To tackle these issues, the authors propose the OPPO framework, which explicitly optimizes multimodal perception through reinforcement learning. OPPO introduces a holistic perception reward to guide the model in recovering fine-grained visual, auditory, and emotional cues, along with a holistic perception loss that applies KL divergence penalties only to modality-specific evidence to suppress cross-modal hallucinations. Notably, this is the first approach to decompose multimodal perception into quantifiable fine-grained cues and optimize them explicitly. The method leverages reinforcement learning, KL regularization, modality masking, and fine-grained reward modeling, achieving state-of-the-art performance on MER-UniBench and MME-Emotion, and significantly improving utilization and faithfulness scores on the newly introduced MEP-Bench benchmark.

Abstract

We find that current emotion-oriented Omni-MLLMs still lack reliable omni-modal perception: they (i) underutilize multimodal cues in their reasoning trajectories and (ii) exhibit unfaithful behavior, often hallucinating modality-specific statements from other modalities. Building on these insights, we propose OPPO (Omni-Perception Policy Optimization), a reinforcement learning framework that explicitly optimizes multimodal perception. First, an Omni-Perception Reward decomposes ground-truth reasoning into fine-grained visual, acoustic, and emotion cues and rewards trajectories that semantically recover these cues. Second, an Omni-Perception Loss compares the policy under full and unimodally masked inputs, applying a KL penalty only to modality-specific evidence tokens to suppress cross-modal hallucination. We further introduce MEP-Bench, a diagnostic benchmark that quantifies utilization and faithfulness. Experiments show that OPPO achieves state-of-the-art performance on MER-UniBench and MME-Emotion, while substantially improving utilization and faithfulness scores on MEP-Bench, highlighting the importance of sufficient and faithful omni perception for multimodal emotion reasoning.

Problem

• multimodal emotion reasoning
• omni-modal perception
• cross-modal hallucination
• perception faithfulness
• multimodal cue utilization

Innovation

• Omni-Perception Policy Optimization
• multimodal emotion reasoning
• cross-modal hallucination
• reinforcement learning
• MEP-Bench