When Visual Signals Mislead: A Mechanistic Study of Attribute Hallucination in Vision-Language Models

2026-08-11

AI Summary

This work addresses attribute hallucination in vision-language models—where objects are erroneously described in terms of color, material, or state—and investigates its underlying causes. The authors propose VISOR, a novel framework that, for the first time, identifies missing visual signals rather than linguistic priors as the primary source of such hallucinations at the attribute level. Building on this insight, VISOR introduces a multi-path rectification mechanism that operates without relying on prior suppression assumptions. It integrates strategies including void-image diagnosis (VSNR), calibration, abstention, and targeted visual adaptation to accurately detect and differentiate failure modes between color/state attributes and material attributes. Experiments on Qwen, InternVL, and LLaVA demonstrate that VISOR substantially reduces attribute false-positive rates and significantly enhances model reliability.

Abstract

Attribute hallucination---where vision-language models (VLMs) correctly identify an object but mischaracterize its properties---is prevalent yet mechanistically poorly understood. The dominant explanation, language-prior dominance, has motivated prior-suppression methods, but this explanation has not been directly tested at the attribute level. We present VISOR (Visual-Operational Remediation), a unified framework that couples null-image-based diagnosis with routed remediation. Its VSNR diagnostic decomposes each prediction into a visual logit signal and a language-prior signal. Across 10,791 negative-ground-truth samples from three VLM families and three attribute types, the visual signal strongly predicts false positives, whereas the language-prior signal is near chance. VISOR uses this diagnosis to separate two failure modes: low-margin but directionally correct visual signals in color/state attributes, and low-SNR or misaligned visual signals in material attributes. The same diagnosis routes each query to the appropriate operator: calibration for threshold-placement errors, abstention for training-free low-SNR handling, or targeted visual adaptation for material failures that prior suppression cannot correct. Across Qwen, InternVL, and LLaVA, VISOR reduces attribute false positives without relying on the prior-dominance assumption.

Problem

• attribute hallucination
• vision-language models
• visual signals
• language priors
• false positives

Innovation

• attribute hallucination
• vision-language models
• visual signal diagnosis
• VISOR framework
• failure mode routing