From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA

2026-06-29

AI Summary

This study addresses the prevalent issue in traffic video question answering (VideoQA) where high accuracy often stems from models exploiting textual shortcuts rather than genuine visual evidence. Auditing four public VideoQA benchmarks, the authors find that several vision-language models exhibit unchanged or even improved performance when deprived of video inputs. To diagnose and mitigate this shortcut reliance, they introduce three diagnostic tools: two novel dataset-level metrics—Blind Gap and Visual Gain—and an instance-level Shortcut Score that requires no model retraining. These tools enable quantitative assessment and filtering of questions susceptible to textual shortcuts. Experiments demonstrate that the resulting filtered subsets substantially reduce shortcut bias, enhance visual grounding quality, and reveal significant differences across benchmarks in their actual dependence on visual content.

Abstract

High benchmark accuracy does not guarantee genuine use of visual evidence. We study this problem in traffic accident Video Question Answering (VideoQA), where correct answers should depend on scene-specific visual evidence but may instead be inferred from textual shortcuts. Through an audit of four public benchmarks, we find that several recent open-weight Vision-Language Models (VLMs) perform competitively, and sometimes better, without video input. On the MM-AU benchmark, removing video consistently improves accuracy, and adding more frames further degrades performance. To quantify visual dependence, we introduce two dataset-level diagnostics: Blind Gap, measuring above-chance text-only performance, and Visual Gain, measuring the marginal benefit of adding video. We further propose an instance-level Shortcut Score that combines text-only confidence with visual necessity signals, enabling continuous, training-free filtering of shortcut-prone questions. The resulting subsets reduce shortcut bias and improve visual grounding. Our findings reveal large differences in grounding quality across benchmarks and show that visually grounded evaluation, not just high accuracy, is essential in safety-critical VideoQA.

Problem

• VideoQA
• modality collapse
• visual dependence
• shortcut learning
• traffic accident

Innovation

• modality collapse
• visual dependence
• shortcut bias
• VideoQA
• Vision-Language Models