ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

2026-08-13

AI Summary

This study addresses the unreliability of chart reasoning in vision-language models and their dependence on complex supervised data by proposing ChartProbe, a novel diagnostic framework. This approach decomposes reasoning into fundamental skills—perception, localization, and computation—and employs code-generated probes to decouple and independently fine-tune these components, thereby eliminating the need for synthesizing complex reasoning data. Experiments demonstrate that training solely on these foundational skills enables zero-cost transfer to unseen complex tasks across three open-source models. Furthermore, the method achieves significant performance improvements on out-of-distribution benchmarks, including ChartQA and CLEVR. These results validate the effectiveness of composing basic skills for facilitating robust complex reasoning without reliance on elaborate supervision.

Abstract

Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements (\emph{perception}), locating them and binding them to their labels (\emph{grounding}), and performing single-step computations such as ranking, totals, and differences (\emph{simple reasoning}). We introduce \textbf{ChartProbe}, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type (pie charts), a human-written benchmark disjoint from our images and templates (ChartQA), and a non-chart visual domain (CLEVR). Complex visual reasoning can therefore improve without complex-reasoning supervision.

Problem

• Vision-language models
• Chart reasoning
• Visual perception
• Grounding
• Diagnostic study

Innovation

• ChartProbe
• Visual Reasoning Diagnosis
• Skill Decomposition
• Zero Complex-Reasoning Supervision
• Code-generated Probes