Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting

2026-06-30

AI Summary

Existing knowledge-based visual question answering (KB-VQA) benchmarks suffer from inflated accuracy metrics due to non-derivable answers, under-constrained questions, and overly simplistic visual scenes, which collectively overestimate models’ true reasoning capabilities. This work proposes the first audit-and-repair protocol tailored for KB-VQA, leveraging structured knowledge alignment analysis and semantic constraint detection to identify and rectify data flaws. Furthermore, it introduces controlled multi-entity visually ambiguous scenes to heighten the challenge of knowledge–vision alignment. Re-evaluation of models on the repaired and enhanced benchmark reveals substantially altered performance trends, exposing biases in prior evaluations and demonstrating that the proposed protocol effectively shifts assessment from superficial matching toward verifiable cross-modal reasoning.

Abstract

Knowledge-Based Visual Question Answering (KB-VQA) aims to evaluate whether Visual Language Models (VLMs) can retrieve, ground, and reason over external structured knowledge beyond visual evidence. In practice, answer accuracy is widely adopted as the primary evaluation metric, implicitly treating correctness as a proxy for knowledge-grounded reasoning. However, for existing KB-VQA benchmarks, this proxy relies on critical assumptions that are often overlooked and rendered unreliable by benchmark issues: annotated answer must be derivable from the associated knowledge base, question must be well-posed with sufficient constraints, and visual setting must meaningfully require grounded disambiguation. In this work, we show that these assumptions are systematically violated in existing KB-VQA benchmarks. Our audit reveals substantial instances with missing or contradicted answers and underspecified questions that render accuracy a misleading metric. Furthermore, we find that existing datasets rely on visually trivial, single-entity scenes that bypass the need for sophisticated visual-to-knowledge mapping. We demonstrate that even with controlled architectures, these flaws lead to distorted model rankings and overestimations of reasoning capabilities. To address this, we introduce (1) a principled audit-and-repair protocol that restores answer derivability and question clarity, and (2) a controlled multi-entity augmentation protocol that introduces visual ambiguity to challenge initial retrieval and grounded reasoning. Re-evaluation under corrected and augmented settings yields markedly different performance trends. Our findings call for rethinking evaluation protocols and designing more interaction-aware KB-VQA benchmarks that prioritize verifiable reasoning over simple matching.

Problem

• Knowledge-Based VQA
• benchmark flaws
• answer derivability
• visual ambiguity
• reasoning evaluation

Innovation

• KB-VQA
• benchmark auditing
• answer derivability
• visual ambiguity
• grounded reasoning