Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis

2026-08-17

AI Summary

This study addresses the challenges of cross-temporal-scale alignment and fine-grained sentiment loss in LLM-based multimodal sentiment analysis by proposing the MGSI framework. The method refines non-textual features through multi-granularity temporal encoding and text-guided alignment, incorporating a polarity-intensity-aware enhancement mechanism to compress multimodal information into pseudo-tokens that efficiently drive frozen LLMs. Experimental results demonstrate that MGSI significantly outperforms frozen-LLM baselines across four benchmarks while achieving performance comparable to strong multimodal approaches. Furthermore, comprehensive ablation studies validate the effectiveness of each module. Collectively, this work establishes a novel paradigm for parameter-efficient multimodal sentiment analysis, effectively bridging temporal misalignment and preserving nuanced emotional cues without requiring extensive model fine-tuning.

Abstract

Multimodal sentiment analysis (MSA) aims to predict sentiment polarity and intensity from heterogeneous inputs such as text, audio, and vision. While large language models (LLMs) offer strong semantic priors for MSA, effectively incorporating audio and visual signals effectively remains challenging. A key challenge is that audio and visual sentiment cues evolve over different temporal scales, yet many LLM-based methods compress these signals through shallow projection or coarse pooling before fusing them with text, which can weaken cross-modal alignment and erase fine-grained affective information. We propose MGSI, a multi-granularity sentiment integration framework for LLM-based MSA. MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends. It then refines non-text features through text-guided alignment, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples. The resulting multimodal representation is finally compressed into a small set of pseudo-tokens for efficient conditioning of a frozen LLM. Experiments on four public benchmarks show that MGSI substantially outperforms frozen-LLM baselines and remains competitive with strong multimodal methods. Further ablation and sensitivity analyses support the effectiveness of multi-granularity temporal modeling, text-guided refinement, and adaptive sentiment calibration.

Problem

• Multimodal Sentiment Analysis
• Large Language Models
• Cross-modal Alignment
• Temporal Scales
• Fine-grained Affective Information

Innovation

• Multi-Granularity Temporal Modeling
• Text-Guided Alignment
• Polarity-Intensity Awareness
• Pseudo-Token Compression
• Frozen LLM Conditioning