From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making

2026-09-04

AI Summary

研究通过在视频-文本注意路径上进行层级因果干预，探讨视觉信息如何影响基于语言的决策，揭示了名词和动词在多模态推理中的不同作用及模型处理时间关系时的挑战。

Abstract

Vision-Language Models are commonly evaluated through their final predictions, but understanding whether these decisions are grounded in visual evidence requires tracing how visual information contributes to language-based decisions. With this purpose in mind, we investigate cross-modal information flow in a video-based generative multiple-choice-like setting by applying a layer-wise causal intervention on video-text attention pathways. We target spatial, causal, and temporal visual reasoning. Our results show that visual information is mainly integrated while the model processes the candidate answer options, which serve as the primary textual grounding sites for the final decision. We further show that nouns play an important role as semantic anchors during multimodal enrichment, while verbs are more relevant when temporal relations are processed. Finally, we identify a distinct pattern in temporal reasoning, suggesting that VLMs struggle to reconstruct sequential information across video frames, but we remark that such fragility may also reflect linguistic biases associated with specific temporal expressions used for defining the relation between events within a scene.

Problem

• Vision-Language Models
• multimodal decision-making
• visual information flow
• cross-modal information

Innovation

• layer-wise causal intervention
• visual-textual information flow
• multimodal decision-making
• temporal reasoning