HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

2026-04-07

AI Summary

This work addresses the prevalent issue of object hallucination in vision-language models during image captioning, where existing attention-based detection methods prove unreliable due to confounding factors such as spatial position and token repetition. To overcome this limitation, the authors propose HaloProbe, a novel Bayesian framework that decouples external descriptive statistics (priors) from internal decoding signals (evidence)—a first in the field. This approach effectively uncovers Simpson’s paradox inherent in coarse-grained attention analyses and enables accurate estimation of token-level hallucination probabilities. By incorporating non-invasive decoding guidance, HaloProbe significantly reduces object hallucination without compromising caption fluency or utility, outperforming state-of-the-art intervention strategies.

Abstract

Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. We reveal that coarse-grained attention-based analysis is unreliable due to hidden confounders, specifically token position and object repetition in a description. This leads to Simpson's paradox: the attention trends reverse or disappear when statistics are aggregated. Based on this observation, we introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe uses balanced training to isolate internal evidence and combines it with learned prior over external features to recover the true posterior. While intervention-based mitigation methods often degrade utility or fluency by modifying models' internals, we use HaloProbe as an external scoring signal for non-invasive mitigation. Our experiments show that HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art intervention-based methods while preserving utility.

Problem

• object hallucinations
• vision-language models
• hallucination detection
• hallucination mitigation
• Bayesian detection

Innovation

• Bayesian hallucination detection
• vision-language models
• object hallucinations
• non-invasive mitigation
• Simpson's paradox