Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents

2026-06-12

AI Summary

This work addresses a critical limitation in existing GUI agents, which rely on full-screen screenshots as visual memory—a practice that mitigates state misinterpretation but often leads to action execution failures due to hidden interaction regions and grounding errors. The study introduces the first comprehensive taxonomy of four failure modes within the perception–reasoning–action pipeline of GUI agents and proposes AGMem, a novel visual memory framework grounded in action execution. Instead of storing entire screenshots, AGMem retains only local image regions associated with successful actions or recovery behaviors. By integrating action-region cropping, visual memory retrieval, and action alignment, AGMem significantly enhances decision accuracy. Evaluated on the OSWorld benchmark, AGMem improves task success rates by 33.3% over full-image memory baselines and substantially reduces action-level failure rates.

Abstract

Graphical User Interface (GUI) agents are increasingly used to automate complex computer tasks across applications, websites, and operating systems. To improve their reliability, recent work has introduced experiential memory, where agents retrieve prior trajectories to guide decision-making in similar states. More recent approaches further extend this idea to visual memory by storing and retrieving screenshots from past interactions, providing agents with richer contextual information than text-only memories. However, the effect of visual memory in GUI agents remains insufficiently understood: it is unclear which failures visual memory mitigates, or which failures it exacerbates. To systematically analyze the effect of visual memory, we introduce a taxonomy of four GUI agent failures (i.e., cognitive failure, visual state misunderstanding, hidden operation blindness, and grounding error) that map to distinct stages of the perception-reasoning-action pipeline. We find that prepending full-image memory has a divergent effect on the failure distribution: it reduces state-level failures but worsens action-level ones, and increases hidden operation blindness and grounding error. Motivated by this finding, we propose Action-Grounded Visual Memory (AGMem), an action-grounded memory framework for GUI agents. The core idea of AGMem is to store image crops that capture the local GUI region closely related to a successful action or a recovery, rather than storing full screenshots. Experiments on OSWorld show that AGMem improves task success rates by 33.3 % over full-image memory. These results demonstrate that AGMem is an effective representation for visual memory in GUI agents.

Problem

• GUI agents
• visual memory
• failure modes
• memory representation
• grounding error

Innovation

• Action-Grounded Visual Memory
• GUI agents
• visual memory
• failure-mode analysis
• memory grounding