VADER: Adaptive Debiasing for Hallucination Mitigation in Video Large Language Models

2026-08-09

AI Summary

This work addresses the prevalent issue of hallucinated responses in video large language models (VideoLLMs) during open-ended video understanding, where generated answers often lack grounding in visual evidence. To mitigate this without requiring additional training, the authors propose an adaptive, training-free debiasing framework that dynamically reweights visual evidence to suppress hallucinations. The approach integrates cross-layer diagnosis of vision–text evidence flow, pre-softmax attention redistribution, masking of high-importance visual tokens, and contrastive decoding to enable fine-grained, frame-wise visual focus intervention. Experimental results demonstrate that the framework significantly improves event-level localization accuracy and temporal consistency across multiple VideoLLMs, achieving a 72.60% accuracy on the EventHallusion benchmark with LLaVA-Video-7B.

Abstract

Large vision-language models (LVLMs) have demonstrated strong performance in open-ended video understanding, yet they remain prone to fluent responses unsupported by video evidence. Existing training-free methods typically apply a globally fixed visual intervention or construct a contrastive branch through input perturbation. The former cannot accommodate video-dependent fusion paths, while the latter can be compensated by cross-frame redundancy. We therefore propose Video-Adaptive Debiasing via Evidence Reweighting (VADER), a training-free framework with two complementary modules. Visual Focus Reallocation (VFR) automatically instantiates an intervention policy for each video-question input: it diagnoses layer-wise visual-to-text evidence flow, determines where to intervene, and derives how strongly to reallocate pre-softmax attention from system-token to video-token blocks. Selective Evidence Erasure (SEE) independently masks high-importance visual tokens in every frame, constructing a prior-biased branch that is difficult to compensate through neighboring frames. Contrastive decoding then down-weights predictions that remain confident after selective evidence erasure. Across multiple VideoLLMs, VADER yields substantial improvements on event-level grounding and temporal consistency; on LLaVA-Video-7B, it reaches 72.60% accuracy on EventHallusion.

Problem

• hallucination mitigation
• video large language models
• evidence grounding
• temporal consistency
• visual-language alignment

Innovation

• adaptive debiasing
• evidence reweighting
• visual focus reallocation
• selective evidence erasure
• contrastive decoding