No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs

2026-06-30

AI Summary

This work addresses the limitation of existing video foundation models in hallucination evaluation, particularly their inability to control for visual background consistency and accurately attribute error sources. To this end, we introduce VidPair-Halluc, a novel benchmark featuring the first background-controlled adversarial video pair design. By leveraging highly similar backgrounds paired with semantically distinct foregrounds, our approach enables precise detection and attribution of model hallucinations. Built upon the PairFlow pipeline and integrating text-to-image and video generation techniques, we systematically construct 1K high-quality adversarial video pairs and 11K spatiotemporal question-answer pairs spanning ten semantic dimensions of spatiotemporal reasoning. Experimental results reveal that state-of-the-art video foundation models still face significant challenges in fine-grained understanding.

Abstract

We introduce VidPair-Halluc, a new benchmark for evaluating video hallucination in large video models (LVMs) under rigorous and controlled conditions. Unlike previous benchmarks that primarily rely on text-based perturbations or adversarial questions while neglecting the consistency of visual backgrounds, VidPair-Halluc features video pairs with highly similar backgrounds but distinctly different foreground semantics, enabling precise attribution of model errors to genuine hallucination rather than background variation. The benchmark is constructed through PairFlow, a pipeline that leverages recent advances in text-to-image and video generation to systematically compose stories, generate coherent video clips, and assemble them into adversarial pairs. Covering both spatial and temporal reasoning across ten semantic aspects, VidPair-Halluc comprises 1K high-quality adversarial video pairs and 11K spatio-temporal QA pairs with control over background and foreground variations. Evaluations on mainstream LVMs show persistent difficulty with robust fine-grained video understanding in adversarial settings, and code and data are available at the https://jethrojames.github.io/VidPair-Halluc/.

Problem

• video hallucination
• large video models
• background consistency
• adversarial evaluation
• spatio-temporal reasoning

Innovation

• video hallucination
• adversarial video pairs
• background-controlled benchmark
• spatio-temporal reasoning
• large video models