Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysis

2026-09-06

AI Summary

本文针对多模态情感分析中的文本与视觉信息融合问题，提出了一种视觉引导的文本提示调优方法(VG-TPT)，通过可控的视觉校准冻结文本表示，有效提高了情感分析性能。

Abstract

Multimodal sentiment analysis requires effective modeling of both verbal semantics and non-verbal affective cues. A central challenge is to calibrate text-centered sentiment understanding with visual facial evidence in a controlled, adaptive, and parameter-efficient manner. Text usually serves as the semantic anchor, whereas visual cues provide complementary evidence for ambiguous or implicit expressions; however, indiscriminate fusion may introduce visual noise and distort textual semantics. Moreover, fully fine-tuning large visual and textual encoders is costly and prone to overfitting on limited and scenario-dependent MSA benchmarks. To address these issues, we propose Vision-Guided Text Prompt Tuning (VG-TPT), which formulates visual-text sentiment modeling as controllable visual calibration of frozen text representations. VG-TPT injects visual affective cues into a frozen BERT encoder through layer-wise adaptive prompts, rather than relying on late-stage feature fusion or full backbone tuning. A co-guided router composes prompts from a trainable prompt bank according to both the evolving text state and the visual guidance feature, enabling sample-specific and layer-specific modulation. Experiments on CMU-MOSEI and CMU-MOSI show that VG-TPT consistently improves over text-only baselines and achieves competitive or superior performance compared with several full-modality methods, while updating only 2.4M trainable parameters. The code is available at https://github.com/ma-tubu/VG-TPT.

Problem

• multimodal sentiment analysis
• textual semantics
• visual cues
• parameter-efficient
• overfitting

Innovation

• Vision-Guided Text Prompt Tuning
• layer-wise adaptive prompts
• co-guided router
• frozen BERT encoder
• multimodal sentiment analysis