Problem
Research questions and friction points this paper is trying to address.
multimodal sentiment analysis
modality reliability
uncertainty
fusion
modality quality
MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis
🤖 AI Summary
This work addresses the challenge in multimodal sentiment analysis that different modalities exhibit significantly varying reliability at the utterance level due to factors such as occlusion, noise, or transcription errors, which can adversely affect traditional fusion approaches. To mitigate this issue, the authors propose MRUF, a method that enables adaptive fusion through multi-granularity routing and uncertainty-aware calibration. Specifically, MRUF incorporates modality importance supervision derived from leave-one-out error increases, an inverse-variance reweighting mechanism for uncertainty-based gating, and a modality-invariant contrastive alignment strategy. Experimental results demonstrate that MRUF consistently outperforms strong baselines on both aligned and unaligned settings of the CMU-MOSI and CMU-MOSEI datasets, effectively down-weighting contributions from high-uncertainty modalities and thereby enhancing model robustness.
📝 Abstract
Multimodal sentiment analysis relies on language, visual, and acoustic cues, but utterance-level modality quality may vary due to occlusion, background noise, motion blur, or imperfect transcripts, causing conventional fusion to over-trust unreliable modalities. We propose MRUF, a reliability-aware fusion method that combines multi-granularity routing with uncertainty-aware calibration. MRUF summarizes sentiment-relevant representations, performs subspace- and modality-level routing, and supervises modality routing with leave-one-out error increases to estimate utterance-level modality importance. It further predicts modality-wise uncertainty and refines modality gates through inverse-variance reweighting, while modality-invariant contrastive alignment stabilizes the shared representation space. Experiments on CMU-MOSI and CMU-MOSEI under aligned and unaligned settings show consistent improvements over strong baselines, and mechanism analysis verifies that modalities with higher predicted uncertainty receive lower fusion weights.
Innovation
Methods, ideas, or system contributions that make the work stand out.
uncertainty-aware fusion
multi-granularity routing
modality reliability
inverse-variance reweighting
contrastive alignment
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
