Towards Interpreting Visual Information Processing in Vision-Language Models

2024-10-09

arXiv.org

AI Summary

This work investigates how visual tokens are processed within the language model component of vision-language models (VLMs). Focusing on the LLaVA architecture, we propose a three-tier interpretability framework: (1) Through hierarchical token attribution and representation alignment visualization, we first observe that visual token representations progressively align with the textual vocabulary space across layers; (2) We identify critical semantic information concentrated in the final visual token, revealing a fact-retrieval–like predictive mechanism analogous to pure language models; (3) Systematic ablation experiments demonstrate that removing object-specific visual tokens degrades recognition accuracy by over 70%, while representational interpretability markedly increases with network depth. Our study establishes the first fine-grained, cross-layer, token-level interpretability paradigm for vision–language fusion in multimodal models, offering novel mechanistic insights into how visual information is encoded, aligned, and utilized within language decoders.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-Language Models (VLMs) are powerful tools for processing and understanding text and images. We study the processing of visual tokens in the language model component of LLaVA, a prominent VLM. Our approach focuses on analyzing the localization of object information, the evolution of visual token representations across layers, and the mechanism of integrating visual information for predictions. Through ablation studies, we demonstrated that object identification accuracy drops by over 70% when object-specific tokens are removed. We observed that visual token representations become increasingly interpretable in the vocabulary space across layers, suggesting an alignment with textual tokens corresponding to image content. Finally, we found that the model extracts object information from these refined representations at the last token position for prediction, mirroring the process in text-only language models for factual association tasks. These findings provide crucial insights into how VLMs process and integrate visual information, bridging the gap between our understanding of language and vision models, and paving the way for more interpretable and controllable multimodal systems.

Problem

• Understanding visual token processing in LLaVA's language model
• Analyzing object localization and visual token representation evolution
• Exploring visual-textual integration for prediction in VLMs

Innovation

• Analyzes visual token localization in LLaVA
• Tracks visual token evolution across layers
• Extracts object info at last token position