Inside VLM Chart Reading: Tracing Value Reading from Vertical Bar Charts Across Space and Depth

2026-09-12

AI Summary

研究通过控制反事实激活修补方法分析了视觉-语言模型在读取垂直条形图数值时的内部计算机制，揭示了模型如何结合几何和尺度状态来恢复准确值。

Abstract

Vision--language models (VLMs) can answer chart questions accurately, but output accuracy does not show how they combine the evidence needed to recover an exact value. We study vertical-bar value reading with controlled counterfactual activation patching in Qwen2.5VL-7B-Instruct and InternVL3.5-8B. The study connects three analyses: (1) The single-factor results show that the changed bar-top region restores much more answer preference than the unchanged bar body, despite containing fewer visual tokens. Legend- and series-related states also lose local recoverability earlier than bar-geometry and axis-scale states. (2) In the handoff analysis, restoration shifts from visual legend regions in early layers to prompt-series positions in middle layers. Resetting the prompt-series state selectively reduces legend-source rescue, supporting its role as a partial mediator. (3) In the factorial analysis, both models can use geometry and scale states from separate donors to favor the combined target. InternVL performs similarly when the states come from separate donors or one image, while Qwen shows lower restoration for separate donors which suggests higher context sensitivity. Together, these results provide preliminary causal evidence for localizing the internal computations that support exact bar-value reading.

Problem

• Vision-language models
• vertical bar charts
• value reading
• evidence combination
• internal computations

Innovation

• controlled counterfactual activation patching
• vertical-bar value reading
• local recoverability
• internal computations localization
• context sensitivity