Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?

2026-06-30

AI Summary

This work addresses the critical issue that existing vision-language models often produce high-confidence predictions even under visually ambiguous inputs, leading to biased answers. Conventional semantic entropy fails to adequately capture uncertainty due to overconfident visual embeddings. To overcome this limitation, the authors propose Visual Semantic Entropy (VSE), a novel uncertainty quantification method that perturbs only the image while keeping the textual query fixed. VSE estimates visual uncertainty by clustering the semantic prototypes of generated answers and computing a quality-weighted dispersion metric, thereby avoiding confounding effects from text perturbations. This approach is the first to effectively capture visual ambiguity and achieves state-of-the-art performance in uncertainty estimation across five prominent vision-language models and five diverse VQA benchmarks, significantly outperforming existing methods.

Abstract

Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions. Common entropy-based methods, such as Semantic Entropy (SE), rely on output diversity. Yet our analysis shows that overconfident visual embeddings suppress output diversity under stochastic decoding, causing SE to underestimate uncertainty in such cases. Recent methods instead probe output diversity through input perturbations, including textual paraphrasing or joint text-image perturbations, and show improved performance. We study these approaches and reveals that the resulting variability is often dominated by textual changes rather than visual evidence, causing uncertainty estimates to reflect prompt sensitivity rather than visual ambiguity. We therefore propose Visual Semantic Entropy (VSE), which perturbs only the image to probe nearby visual variations while keeping the text query fixed. VSE measures uncertainty by clustering generated answers into semantic prototypes and computing the mass-weighted dispersion among them. Extensive evaluation across five modern vision-language models and five diverse VQA benchmarks demonstrates that VSE effectively captures visual ambiguity, establishing a new state-of-the-art for VLM uncertainty estimation.

Problem

• visual ambiguity
• vision-language models
• uncertainty estimation
• semantic entropy
• visual semantic entropy

Innovation

• Visual Semantic Entropy
• vision-language models
• visual ambiguity
• uncertainty estimation
• image perturbation