What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks

2026-07-11

AI Summary

This study addresses the instability and poor interpretability of short-answer visual question answering (VQA) benchmark evaluations, which often misclassify semantically correct answers as errors due to overreliance on superficial string matching. Leveraging a high-precision (97.6%) human-validated semantic judgment protocol, the authors conduct a systematic audit of over 37k official errors from six multimodal models across six benchmarks, revealing that nearly half of these “errors” are in fact semantically accurate but differ in surface form. Through text-only model replication, deterministic CPU-based repair contracts, and answer-type diagnostics, the work demonstrates that evaluation bias stems from the scorer’s preference for lexical form over meaning, and shows that simple prompting or contextual fine-tuning can substantially improve scoring stability. The study advocates semantic auditing and answer-type analysis as essential complements to standard VQA benchmark evaluation.

Abstract

Short-answer VQA benchmarks conflate two distinct quantities: whether a model's answer is semantically correct, and whether that answer matches the surface form expected by the automatic evaluator. We study this conflation across six vision--language models and six benchmarks, using a human-validated semantic judge (97.6% precision) to audit over 37k official errors. A second text-only judge reproduces the same benchmark-level false-negative pattern, showing that the effect is not an artifact of a single audit model. On text-rich benchmarks, up to half of these errors are semantically acceptable answers penalized purely for surface-form mismatch. This instability is structured by answer type: extractive and multi-span answers are far more evaluator-sensitive than scalar answers. Benign prompt and context rewrites further destabilize official outcomes, flipping item-level correctness at substantial rates without changing the underlying task. A deterministic CPU-only contract repair confirms that the undercount is partially recoverable. These findings imply that official short-answer VQA scores should be accompanied by semantic audits and answer-type diagnostics to remain interpretable.

Problem

• short-answer VQA
• evaluator-dependent instability
• semantic correctness
• surface-form mismatch
• multimodal benchmarks

Innovation

• evaluator-dependent instability
• semantic correctness
• surface-form mismatch
• short-answer VQA
• answer-type sensitivity