Do Vision Language Models Need to Process Image Tokens?

2026-04-10

AI Summary

This study investigates whether visual tokens in vision-language models require sustained processing in deep layers. Through analyses of representation entropy, intrinsic dimensionality, and trajectory curvature—combined with depth truncation and deterministic decoding experiments—the authors find that visual representations rapidly converge to a stable state in shallow layers, rendering deeper processing largely inconsequential for final outputs. The work further reveals substantial interchangeability of visual tokens across layers, challenging the prevailing paradigm that multimodal large language models critically depend on deep visual processing. Additional analysis shows that single-token prediction is robust to visual depth truncation, whereas multi-token generation tasks rely on continuous visual input; crucially, deep visual processing primarily shapes the inference trajectory rather than the ultimate result.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Vision Language Models (VLMs) have achieved remarkable success by integrating visual encoders with large language models (LLMs). While VLMs process dense image tokens across deep transformer stacks (incurring substantial computational overhead), it remains fundamentally unclear whether sustained image-token processing is necessary for their performance or visual representations meaningfully evolve from early to later layers. In this work, we systematically investigate the functional role of image tokens in VLMs and show that visual representations rapidly converge to a bounded-complexity regime, \ie their entropy stabilizes, intrinsic dimensionality compresses, and trajectory curvature approaches a near-constant profile. In contrast, textual representations continue to undergo substantial restructuring across depth. Once stabilized, visual representations become largely interchangeable between layers, indicating limited additional transformation in deeper stages. Further, depth-wise visual truncation reveals that the necessity of visual processing is task-dependent, where single-token predictions remain comparatively robust to truncated visual depth, but multi-token generation require sustained access to visual representations. Under deterministic decoding, reducing visual depth perturbs intermediate reasoning trajectories more strongly than final outputs, suggesting that image tokens influence the structure of reasoning more than the ultimate conclusions. Collectively, these findings \textbf{question the assumption} that deeper visual processing is uniformly essential in VLMs, challenging the current paradigm of multimodal LLM architectures.

Problem

• Vision Language Models
• image tokens
• visual representations
• computational overhead
• multimodal LLMs

Innovation

• vision-language models
• image token processing
• representation convergence
• depth-wise truncation
• multimodal reasoning