ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation

2026-05-28

AI Summary

Existing evaluations of multimodal large language models predominantly focus on the manifestations of hallucinations rather than their underlying causes, often employing simplistic scenarios and limited assessment formats. This work proposes ReactBench, the first causality-driven benchmark grounded in the root causes of hallucinations. It constructs four diagnostic tasks—relation ablation, counterfactual attributes, image perturbation tracking, and dense counting—by leveraging adversarial images and suggestive queries, and integrates chain-of-thought reasoning to enable fine-grained attribution analysis. Designed in an exam-style evaluation format, ReactBench moves beyond conventional accuracy-only metrics. Experimental results demonstrate that current models remain highly susceptible to specific hallucination triggers, underscoring ReactBench’s effectiveness in diagnosing model weaknesses and enhancing robustness and interpretability.

Abstract

While multimodal large language models (MLLMs) have achieved rapid progress in vision-language understanding, they remain prone to multimodal hallucinations, producing responses that are inconsistent with the visual input. Existing benchmarks predominantly focus on detecting hallucination outcomes rather than evaluating the underlying causes of these failures. Moreover, many benchmarks rely on simplistic scenarios and limited evaluation formats that no longer challenge state-of-the-art models. To address these limitations, we introduce ReactBench, a cause-driven hallucination benchmark featuring multiple tasks and an exam-style evaluation format. By generating adversarial images and hallucination-inducing queries, ReactBench introduces four targeted tasks: Relational Erasure, Counterfactual Attribute, Alteration Tracing, and Dense Counting. These tasks systematically expose co-occurrence bias, language priors, cross-image comparative perception deficiencies, and fine-grained perceptual bottlenecks. Beyond standard accuracy-based evaluation, we leverage Chain-of-Thought reasoning to identify fine-grained sub-causes of hallucination within each task. Extensive evaluations reveal that current MLLMs remain notably vulnerable to cause-specific hallucination triggers, demonstrating the value of ReactBench as a systematic and interpretable testbed for diagnosing and improving multimodal model robustness. The project page is available at https://reactbench.github.io/.

Problem

• multimodal hallucination
• cause-driven evaluation
• vision-language models
• benchmark limitations
• systematic diagnosis

Innovation

• cause-driven benchmark
• multimodal hallucination
• adversarial evaluation
• Chain-of-Thought reasoning
• systematic diagnosis