Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure

2026-07-05

AI Summary

This work addresses the overreliance of GUI agents on structured information (e.g., DOM) for state understanding, which often leads to erroneous decisions due to insufficient utilization of visual pixels. The authors introduce the concept of “visual state dependency” and a novel metric termed the “perception-fusion gap,” quantifying agent reliance on pixels, structure, and priors through paired single-channel intervention experiments across 310 real-world interface probes. By integrating deterministic forced-choice scoring, white-box ablation, and dual-channel input comparisons, the study provides the first direct diagnosis of the visual grounding of state beliefs. Empirical results reveal that five state-of-the-art models exhibit a positive perception-fusion gap—indicating that their textual state representations implicitly depend on structural cues despite high image recognition accuracy—whereas coordinate-based action agents remain largely unaffected. Notably, structural overreliance can lead to actual task failures.

Abstract

Multimodal GUI agents read an interface through two redundant channels: the rendered pixels of a screenshot and a serialized structure such as a DOM or accessibility tree. Before acting, an agent forms a belief about the current interface state, but existing benchmarks score task success, element grounding, or attack resistance and do not ask whether that belief is drawn from the pixels. We formalize visual state reliance, the attribution of a state belief to pixels, structure, or priors, and measure it with paired single-channel interventions over 310 real web, mobile, and desktop probes. Every probe is scored by deterministic forced choice, with no model-generated item and no model judge. Our central metric is the Perception-Fusion Gap, the fraction of probes a model perceives correctly yet resolves toward structure under conflict. Across five models from three vendors, textual state beliefs defer to structure while image-only accuracy stays near ceiling, and Perception-Fusion Gap is positive for every model; non-text identity, by contrast, stays largely pixel-bound. The substitution is specific to the serialized-text and indexed-action channel, and coordinate-action agents are largely immune. For textual conflicts, a white-box ablation traces the effect to a single copied structural value, and in two live environments the conflict drives wrong actions and real task failure. Visual state reliance therefore gives a measurable diagnostic of whether agent state beliefs are visually grounded, and the errors it exposes propagate to actions.

Problem

• GUI agents
• state-belief reliance
• visual grounding
• Perception-Fusion Gap
• multimodal perception

Innovation

• visual state reliance
• Perception-Fusion Gap
• GUI agents
• multimodal grounding
• structured vs. pixel input