CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering

2026-08-11

AI Summary

This work addresses the limitations of existing image captioning evaluation methods, which struggle to comprehensively verify dense, fine-grained factual content and lack region-aligned coverage of full-scene visual information. To overcome this, the authors reformulate caption evaluation as a dense question-answering task grounded in full-scene semantic regions, introducing the first dense QA benchmark that encompasses both foreground and background elements. Fact-checking is implemented through multiple-choice questions with region alignment, augmented by an innovative uncertainty option and a density-aware metric. A structured scoring mechanism—built upon a two-level domain taxonomy and integrating a language discriminator, effective accuracy, and density penalty—enhances evaluation granularity. Experiments across 13 vision-language models reveal substantial scene coverage gaps and trade-offs between capability and efficiency, uncovering failure modes invisible to conventional sparse or overlapping evaluation paradigms.

Abstract

Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity. Reference-based metrics (e.g., CIDEr and SPICE) and LLM-as-scorer protocols struggle to verify dense factual claims, while existing QA-based alternatives generally offer lower probe density, narrower domain coverage, or no explicit alignment between individual questions and segmented image regions. We introduce CapProbe, a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking. Each image is decomposed into coarse semantic regions covering both foreground and background elements; for every retained region, we generate multiple-choice questions spanning 10 semantic categories, forming a dense checklist of probed visual facts. Guided by a two-tier taxonomy of 37 L1 domains and 219 L2 sub-domains, CapProbe comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image. A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones, while density-based metrics penalize verbose yet uninformative captions. The protocol is cost-effective: by converting unconstrained scalar scoring into structured MCQ reading, it reduces open-ended scoring bias while remaining judge-conditioned and yields relatively stable model rankings under a fixed reader. Experiments on 13 VLMs show large Coverage gaps across models, a clear competency-efficiency trade-off, and failure modes that sparse or overlap-based evaluation often misses. The benchmark data, annotations, and evaluation code will be released soon.

Problem

• image caption evaluation
• dense factual verification
• vision-language models
• region-aligned QA
• benchmarking

Innovation

• dense QA
• region-aligned evaluation
• vision-language models
• factual consistency
• benchmark