DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams

2026-04-28

AI Summary

Current vision-language models often lack explicit reliance on visual evidence to support their answers in chart question answering, leading to poor interpretability and unreliable evaluation. To address this limitation, this work proposes the DRAGON benchmark, which introduces—for the first time—an evidence localization task tailored to chart-based visual reasoning. The benchmark requires models to simultaneously answer questions and identify key visual regions such as labels, legends, and axes. It encompasses six chart types, provides 11,664 fine-grained evidence annotations, and includes a standardized evaluation framework along with a test set of 2,445 human-verified samples. Evaluation of eight state-of-the-art vision-language models reveals their general deficiency in localizing reasoning cues across diverse chart types, underscoring the critical need for evidence-driven reasoning in this domain.

Abstract

Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics, circuit schematics, and scientific diagrams. Recent vision-language models (VLMs) often achieve high answer accuracy on these tasks, yet correct answers do not guarantee that models ground their reasoning in the diagram regions that support the prediction. Models may instead rely on textual correlations or dataset artifacts without identifying the visual evidence required to verify the answer. This limitation prevents reliable evaluation of diagram reasoning and reduces interpretability. We introduce DRAGON, a benchmark for evaluating evidence-grounded visual reasoning in diagrams. Given a diagram, a question, and the correct answer, a model must predict bounding boxes that correspond to the visual elements required to justify the answer. These evidence regions may include answer-bearing components, textual labels, legends, axes, connectors, and other supporting structures involved in the reasoning process. The DRAGON dataset contains 11,664 annotated question instances collected from six diagram QA datasets: ChartQA, Circuit-VQA, InfographicsVQA, MapIQ, MapWise, and AI2D. We release a 2,445-instance benchmark test set with human-verified reasoning evidence annotations and a standardized evaluation framework. We evaluate eight recent VLMs and analyze their ability to localize reasoning evidence across diverse diagram domains. DRAGON enables systematic evaluation of diagram reasoning and supports future research on models that ground their predictions in visual evidence.

Problem

• diagram question answering
• evidence grounding
• visual reasoning
• vision-language models
• interpretability

Innovation

• evidence-grounded reasoning
• diagram question answering
• visual grounding
• vision-language models
• benchmark dataset