DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models

2026-04-18

AI Summary

Current vision-language models are prone to errors in object existence judgment due to either interference from textual priors or insufficient perceptual grounding, yet existing benchmarks lack the capability to disentangle these two failure modes. This work proposes DO-Bench, a diagnostic benchmark that employs a dual-dimensional intervention design—Prior Override and Perception-Limited—combined with localized cropping and contextual control—to attribute object hallucinations specifically to prior dominance, perceptual deficiency, or their interaction. The benchmark introduces two diagnostic metrics: PriorRobustness and PerceptionAbility. Evaluations across multiple open- and closed-source models reveal systematic differences in their robustness to textual priors and reliability in visual perception, thereby overcoming the limitations of conventional holistic accuracy assessments.

Technology Category

• Computer Vision: Language and Vision
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Object level hallucination remains a central reliability challenge for vision language models (VLMs), particularly in binary object existence verification. Existing benchmarks emphasize aggregate accuracy but rarely disentangle whether errors stem from perceptual limitations or from the influence of contextual textual priors, leaving underlying failure mechanisms ambiguous. We introduce DO-Bench, a controlled diagnostic benchmark that isolates these sources through structured multimodal interventions. Rather than evaluating models in unconstrained settings, DO-Bench probes two complementary dimensions: the Prior Override dimension progressively strengthens contextual textual priors while holding visual evidence constant to assess resistance to prior pressure, and the Perception-Limited dimension incrementally enhances visual evidence from full-scene context to localized object crops to measure perceptual grounding strength. This paired design enables attribution of errors to prior suppression, perceptual insufficiency, or their interaction. We further define two diagnostic metrics, PriorRobust and PerceptionAbility, to quantify these behaviors consistently. Evaluations across diverse open- and closed-source VLMs reveal systematic differences in prior sensitivity and perceptual reliability, demonstrating that object hallucination reflects heterogeneous, mechanism dependent failure patterns beyond aggregate accuracy.

Problem

• object hallucination
• vision-language models
• textual priors
• perceptual grounding
• benchmark

Innovation

• object hallucination
• vision-language models
• controlled benchmark
• prior override
• perceptual grounding