VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

2026-05-19

AI Summary

Current visual question answering (VQA) evaluations lack joint verification of answer correctness and its alignment with pixel-level visual evidence, leading to a disconnect between model reasoning and visual grounding. To address this, this work proposes the VISTAQA benchmark, which requires models to answer free-form questions while simultaneously providing precise segmentation masks as visual evidence. The benchmark introduces the GROVE metric, which jointly evaluates answer accuracy and evidence localization quality via geometric mean. VISTAQA encompasses six task categories and six visual domains, featuring expert-annotated pixel-level masks and hallucination-aware samples. Experimental results demonstrate that even state-of-the-art models achieve limited performance under the GROVE metric, revealing a significant gap between answer correctness and faithful alignment with visual evidence.

Abstract

Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning, yet current multimodal large language model (MLLM) evaluations do not explicitly enforce this alignment. Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation, leaving the coupling of reasoning and grounding an open challenge. We introduce VISTAQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering. VISTAQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning. VISTAQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers. It also includes hallucination-aware examples where no valid visual evidence exists. To support this enhanced evaluation, we introduce GROVE, a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other. Comprehensive experiments across grounding-aware models and hybrid pipelines with general-purpose MLLMs reveal that even the strongest systems achieve limited performance under GROVE, highlighting a substantial gap between answer accuracy and visual evidence alignment.

Problem

• visual question answering
• evidence grounding
• multimodal reasoning
• model transparency
• pixel-level alignment

Innovation

• visual question answering
• evidence grounding
• pixel-level segmentation
• multimodal evaluation
• hallucination detection