PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment Analysis

2025-11-16

AI Summary

To address modality competition in multimodal sentiment analysis (MSA), where dominant modalities suppress weaker ones, this paper proposes a prototype-guided collaborative optimization framework. Methodologically: (1) a modality-prototype space is constructed, with prototype alignment and entropy-regularized optimal transport ensuring cross-modal semantic consistency; (2) a prototype-gated fusion module enables dynamic, context-aware modality weighting; (3) a Shapley-value-based gradient modulation mechanism is introduced to fairly allocate modality contributions during backpropagation, mitigating competitive suppression. Evaluated on IEMOCAP, MOSI, and MOSEI, the model achieves significant improvements in sentiment classification accuracy and cross-modal robustness. It establishes new state-of-the-art performance, empirically validating the effectiveness of the proposed mechanisms in fostering representation-level collaboration and balanced multimodal learning.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multimodal Sentiment Analysis (MSA) seeks to understand human emotions by integrating textual, acoustic, and visual signals. Although multimodal fusion is designed to leverage cross-modal complementarity, real-world scenarios often exhibit modality competition: dominant modalities tend to overshadow weaker ones, leading to suboptimal performance.In this paper, we propose PaSE, a novel Prototype-aligned Calibration and Shapley-optimized Equilibrium framework, which enhances collaboration while explicitly mitigating modality competition. PaSE first applies Prototype-guided Calibration Learning (PCL) to refine unimodal representations and align them through an Entropic Optimal Transport mechanism that ensures semantic consistency. To further stabilize optimization, we introduce a Dual-Phase Optimization strategy. A prototype-gated fusion module is first used to extract shared representations, followed by Shapley-based Gradient Modulation (SGM), which adaptively adjusts gradients according to the contribution of each modality. Extensive experiments on IEMOCAP, MOSI, and MOSEI confirm that PaSE achieves the superior performance and effectively alleviates modality competition.

Problem

• Mitigating modality competition in multimodal sentiment analysis systems
• Aligning unimodal representations through prototype-guided calibration learning
• Adaptively adjusting gradients based on Shapley-based modality contributions

Innovation

• Prototype-aligned calibration refines unimodal representations
• Shapley-based gradient modulation adaptively adjusts contributions
• Dual-phase optimization stabilizes multimodal fusion process