Mirage Probes: How Vision Models Fake Visual Understanding

2026-06-11

AI Summary

This work addresses the tendency of vision-language models to generate confident yet hallucinated responses even in the absence of image inputs, thereby distorting evaluation outcomes. The authors propose Mirage Probes, a contrastive probing framework that constructs question variants paired with labels and employs linear probes to analyze activations in residual streams, MLPs, and attention modules. To disentangle lexical confounds, they introduce a Naive Bayes text-only baseline. Their analysis reveals, for the first time, two distinct hallucination mechanisms: pure textual bias and spurious image generation. They further propose the Prior Harnessing Index (PHI) to quantify a model’s reliance on textual priors. Experiments demonstrate that these two hallucination types exhibit markedly different behaviors across benchmarks, and while textual debiasing mitigates textual bias, it fails to eliminate hallucinations rooted in visual representations.

Abstract

Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using Mirage Probes, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from internal activations across residual stream, MLP, post-attention, and attention-head sites in two open-source VLMs. We demonstrate that a Naive Bayes text baseline cannot recover this signal, ruling out surface lexical confounds. Cross-benchmark separability patterns, together with a novel Prior Harnessing Index (PHI) measuring how much a model can answer from text alone, expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model's visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.

Problem

• vision-language models
• visual grounding
• mirage behavior
• textual bias
• spurious images

Innovation

• Mirage Probes
• visual grounding
• vision-language models
• Prior Harnessing Index
• spurious images