Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

2026-07-14

AI Summary

Current video large language models (VLLMs) often achieve high accuracy on benchmarks, yet this performance may not genuinely reflect their visual understanding capabilities. This work proposes the Visual Dependency Gap (VDG), a diagnostic metric that systematically evaluates the performance discrepancy between models operating on original videos versus black-screen inputs, forming a diagnostic spectrum from no visual content to full video. Through comprehensive experiments on benchmarks such as MVBench—incorporating McNemar’s test, frame-sampling ablations, H.264 compression, and multi-model comparisons—the study reveals that most VLLMs maintain surprisingly strong performance even with black screens, that frame diversity is the primary driver of visual gains, and that temporal information contributes minimally. VDG effectively quantifies the degree of visual grounding across both open-source and API-based models, establishing a new standard for evaluating true visual dependency.

Abstract

Benchmark accuracy in video large language models (LLMs) is often treated as evidence of visual understanding. We audit this assumption across twenty models spanning 2-78B parameters and ten architecture families. We introduce the Visual Dependency Gap (VDG), the difference in per-question correctness between original-video and black-screen conditions. Paired McNemar tests on MVBench show that accuracy and visual dependency are separable: models differ on original video (p = 0.0003) but not on black screens (p = 0.53). Across models, task-type rankings are stable: Attribute Perception is strongly visual, whereas Temporal Reasoning approaches the language-only baseline. A diagnostic ladder from black screen to single frame, shuffled frames, and original video reveals that frame diversity supplies most of the visual benefit, while temporal order contributes near-zero accuracy across sixteen open-weight models. An ablation from 0.5 to 24 FPS rules out sparse sampling as the cause. H.264 experiments further show that stable aggregate accuracy conceals bidirectional question-level answer flips. The diagnostic also generalizes to four API-accessed models, whose VDG values range from 0.025 to 0.315. These results motivate VDG as a standard audit for whether video benchmarks measure visually grounded capability. Code is available at https://github.com/JaeLee18/accuracy-without-grounding.

Problem

• video LLM
• visual understanding
• benchmark evaluation
• visual dependency
• grounding

Innovation

• Visual Dependency Gap
• video LLMs
• visual grounding
• benchmark auditing
• temporal reasoning