Learning Shared Sentiment Prototypes for Adaptive Multimodal Sentiment Analysis

2026-04-07

AI Summary

This work addresses the limitations of conventional multimodal sentiment analysis methods, which often compress heterogeneous affective cues into a single representation during early fusion, thereby failing to preserve their intrinsic structures and lacking the capacity for dynamic modality adjustment during inference. To overcome these issues, the authors propose PRISM, a novel framework that constructs a shared affective prototype space to enable structured cross-modal sentiment representation and comparison. Furthermore, PRISM adaptively reweights the contribution of each modality during inference, moving beyond static fusion paradigms. Extensive experiments on three benchmark datasets demonstrate that PRISM significantly outperforms existing state-of-the-art baselines, confirming its effectiveness and superiority in multimodal sentiment analysis.

Abstract

Multimodal sentiment analysis (MSA) aims to predict human sentiment from textual, acoustic, and visual information in videos. Recent studies improve multimodal fusion by modeling modality interaction and assigning different modality weights. However, they usually compress diverse sentiment cues into a single compact representation before sentiment reasoning. This early aggregation makes it difficult to preserve the internal structure of sentiment evidence, where different cues may complement, conflict with, or differ in reliability from each other. In addition, modality importance is often determined only once during fusion, so later reasoning cannot further adjust modality contributions. To address these issues, we propose PRISM, a framework that unifies structured affective extraction and adaptive modality evaluation. PRISM organizes multimodal evidence in a shared prototype space, which supports structured cross-modal comparison and adaptive fusion. It further applies dynamic modality reweighting during reasoning, allowing modality contributions to be continuously refined as semantic interactions become deeper. Experiments on three benchmark datasets show that PRISM outperforms representative baselines.

Problem

• Multimodal Sentiment Analysis
• Sentiment Prototypes
• Modality Fusion
• Dynamic Reweighting
• Cross-modal Comparison

Innovation

• shared sentiment prototypes
• adaptive multimodal fusion
• dynamic modality reweighting
• structured affective representation
• multimodal sentiment analysis