Wiener Representation Filtering for VLM Hallucination Suppression

2026-08-08

AI Summary

This work addresses the prevalent issue of hallucination in vision-language models (VLMs), where generated text often misaligns with visual input. The authors propose a training-free, post-hoc representation editing method that adapts Wiener filtering principles to suppress hallucinations in VLMs for the first time. By offline calibration, the approach estimates the covariance structures of truthful and hallucinatory components in the feature space and applies closed-form optimal gains derived from empirical second-order statistics to directly correct representations at feedforward layer outputs. Evaluated on models such as LLaVA-1.5 and MiniGPT-4, the method significantly reduces object hallucination across CHAIR, POPE, and MME benchmarks while preserving linguistic fluency. Furthermore, it demonstrates strong generalization capabilities in video understanding and diffusion-based language models.

Abstract

Vision-language models (VLMs) excel at open-ended captioning and visual QA but often describe objects, attributes, or relations absent from the image, a phenomenon known as object hallucination. We propose a {training-free, post-hoc representation editing technique} that operates in the representation space of the language backbone. The method performs a lightweight, one-time offline calibration on a modest paired dataset to estimate the required covariance structures, using only forward passes and empirical second-order statistics with no gradient updates or fine-tuning, after which the correction is absorbed directly into the model's existing weights. By modeling hidden states as a superposition of truthful and hallucination-associated components, we derive a Wiener-type estimator whose optimal gains are given in closed form from the covariances of paired truthful and hallucinated representations. An eigendecomposition yields mode-wise attenuation that respects a stability criterion, i.e., the filter responds continuously to estimation noise. The correction is applied once to the feed-forward output projections of selected deeper layers, at inference time, the model runs unchanged and at the same speed. Experiments on LLaVA-1.5, MiniGPT-4, Gemma3, and mPLUG-Owl2 demonstrate consistent reductions in object hallucination on CHAIR, POPE, and MME while maintaining caption fluency and overall response quality. We further demonstrate the generality of our approach on the TempCompass video understanding benchmark and on discrete diffusion language models for grounded dialogue, showing that representation filtering reduces hallucinations even in temporal video reasoning and multi-step, sequence-wide denoising settings.

Problem

• vision-language models
• object hallucination
• hallucination suppression
• representation filtering
• model reliability

Innovation

• Wiener filtering
• hallucination suppression
• representation editing
• training-free calibration
• covariance estimation