DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?

2026-05-28

AI Summary

Current vision-language models (VLMs) struggle to perceive fine-grained visual differences in web user interfaces, limiting their applicability in scenarios such as GUI agents. To address this, this work proposes DiffSpot—the first controllable and scalable “spot-the-difference” benchmark tailored for web interfaces. DiffSpot generates image pairs by modifying a single CSS attribute in HTML and evaluates models through re-rendering and pixel-level difference localization. The benchmark incorporates difficulty tiers and no-difference samples to mitigate hallucination, revealing that model performance strongly correlates with CSS attribute type rather than pixel-wise or CLIP-based distances. Evaluation of 13 state-of-the-art VLMs on 4,400 image pairs shows that even the best model detects only 40.7% of actual changes, with recall rates below 23% on high-difficulty tasks, underscoring a significant gap in current VLMs’ fine-grained visual perception capabilities.

Abstract

Vision-language models (VLMs) have made strong progress on high-level image-text alignment, yet their ability to perceive subtle visual differences remains limited. We study this problem in rendered web interfaces, where localized visual changes are both a diagnostic test of fine-grained perception and a practical requirement for GUI agents and design tools. We introduce \textbf{DiffSpot}, a code-driven benchmark for open-ended spot-the-difference on web interfaces. DiffSpot constructs controlled image pairs by mutating a single CSS property of a target element in self-contained HTML, re-rendering the page, and recording the changed property, element, and mutation magnitude. A grounding gate retains only pairs whose rendered pixel difference is confined to the target element. The benchmark contains 4{,}400 pairs, including 3{,}900 has-diff pairs balanced across 13 CSS-property operators and three difficulty tiers, plus 500 no-diff pairs for hallucination control. Evaluating 13 frontier VLMs zero-shot, we find that even the best model identifies only $40.7\%$ of true changes, with Hard-tier Recall below $23\%$ for every model. DiffSpot further shows that difficulty is strongly property-dependent: across CSS operators, neither pixel magnitude nor CLIP distance reliably predicts Recall.

Problem

• fine-grained visual differences
• vision-language models
• web interfaces
• spot-the-difference
• CSS property changes

Innovation

• fine-grained visual difference
• vision-language models
• web interface
• CSS mutation
• spot-the-difference benchmark