The Hidden Evolution of Disguised Visual Context inside the VLM

2026-06-18

AI Summary

This study addresses the lack of systematic investigation into how visual tokens evolve and interact with the language space across different integration architectures in contemporary vision-language models. Under unified training conditions, the authors compare context injection and interlayer injection paradigms, revealing for the first time that visual tokens—introduced into large language models as disguised contextual cues—progressively evolve according to the integration strategy and capture distinct frequency-domain features. Through controlled ablation studies, multimodal benchmark evaluations, attention analysis, and cross-layer representation quality assessments, the work demonstrates that model performance hinges on the quality of visual representations at each layer rather than solely on attention allocation, and further elucidates fundamental differences between the two paradigms in terms of task performance, feature utilization, and alignment with linguistic semantics.

Abstract

Visual tokens enter Large Language Models (LLMs) as raw, foreign signals. How they are transformed into meaningful representations and interact with the language space depends entirely on the integration architecture. Whether by treating visual tokens as in-context prompts within the input sequence or injecting them directly into the LLM's intermediate layers. A controlled comparison and understanding of how these architectural choices affect visual information and its internal transformation to integrate with the LLM remains underexplored. We provide a fair comparison by evaluating in-context and layer-wise injection VLM integration paradigms under identical training conditions across single image, multi-image, and video benchmarks. In doing so, we uncover a hidden evolution where visual tokens enter the LLM as disguised visual context, raw representations lacking linguistic structure, but are progressively reshaped depending on the integration paradigm, each capturing fundamentally different frequency characteristics of the visual signal. We show that this evolution inside the LLM determines what visual features the VLM can utilize effectively, how visual representations align with the language space, and ultimately how each paradigm performs across different tasks. We further demonstrate that attention allocation alone is insufficient, and that performance is driven by the quality of visual representations at each layer.

Problem

• visual-language models
• integration architecture
• visual tokens
• representation evolution
• language space alignment

Innovation

• visual-language models
• integration paradigms
• hidden evolution
• visual token transformation
• multimodal representation