When Vision Speaks for Sound

2026-05-13

AI Summary

This work addresses the "Clever Hans effect" in existing audio-visual multimodal large language models, which often hallucinate audio based on visual cues rather than genuinely comprehending auditory content. To systematically probe whether models achieve authentic audio-visual alignment, the authors propose the Thud framework, introducing three counterfactual audio-editing interventions—Shift, Mute, and Swap—for the first time. They further design a two-stage alignment training strategy that integrates preference pairs derived from these interventions with event-level video preference regularization to enhance audio grounding. Evaluated on a 10K-sample training set, the model demonstrates a 28-percentage-point average accuracy improvement across the three intervention dimensions and achieves consistent, albeit modest, performance gains on standard video and audio-visual question-answering benchmarks.

Abstract

Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.

Problem

• audio-visual alignment
• Clever Hans effect
• multimodal learning
• video understanding
• audio grounding

Innovation

• audio-visual alignment
• Clever Hans effect
• intervention-driven probing
• counterfactual audio editing
• multimodal LLMs