Vision-Language Models Can't See the Obvious

2025-07-07

AI Summary

This work identifies a critical deficiency in large vision-language models (LVLMs): their inability to reliably perceive human-salient low-level visual features—such as color, luminance, and orientation. To address this gap, the authors introduce SalBench, the first benchmark explicitly designed to evaluate visual saliency perception. It comprises three novel tasks—Odd-One-Out Detection, Referring Odd-One-Out, and Visual Referring Odd-One-Out—built upon manually annotated high-saliency images. SalBench provides the first systematic, quantitative assessment of LVLMs’ capacity to detect overt visual anomalies. Experiments reveal a substantial performance gap between current models and human observers: even the state-of-the-art GPT-4o achieves only 47.6% accuracy. This work not only exposes a fundamental weakness in LVLMs’ foundational visual perception but also motivates alignment of model attention mechanisms with human visual saliency principles. SalBench serves as both a critical evaluation tool and a catalyst for advancing visual representation learning and multimodal alignment research.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present Saliency Benchmark (SalBench), a novel benchmark designed to assess the capability of Large Vision-Language Models (LVLM) in detecting visually salient features that are readily apparent to humans, such as a large circle amidst a grid of smaller ones. This benchmark focuses on low-level features including color, intensity, and orientation, which are fundamental to human visual processing. Our SalBench consists of images that highlight rare, unusual, or unexpected elements within scenes, and naturally draw human attention. It comprises three novel tasks for evaluating the perceptual capabilities of LVLM: Odd-One-Out Detection, Referring Odd-One-Out, and Visual Referring Odd-One-Out. We perform a comprehensive evaluation of state-of-the-art LVLM using SalBench and our findings reveal a surprising limitation: LVLM struggle to identify seemingly obvious visual anomalies, with even the advanced GPT-4o achieving only 47.6% accuracy on such a simple task. SalBench will be an important step in measuring the capabilities of LVLM that align with the subtle definition of human attention.

Problem

• Assessing LVLM's ability to detect visually salient human-obvious features
• Evaluating LVLM performance on low-level visual features like color and orientation
• Identifying LVLM limitations in recognizing simple visual anomalies

Innovation

• SalBench benchmark tests LVLM salient feature detection
• Focuses on color intensity orientation low-level features
• Three novel tasks evaluate LVLM perceptual capabilities