AnchorGUI: Asymmetric Memory for Dual-Scale Learning in GUI Navigation

2026-09-14

AI Summary

为解决GUI导航中视觉历史处理难题，提出AnchorGUI框架，通过不对称记忆机制实现双尺度学习，提高错误修正和经验提炼效率。

Abstract

Vision-Language Models (VLMs) enable autonomous GUI navigation, but agents still struggle to process and learn from dense, continuous visual histories. This bottleneck hinders both immediate error correction within a single episode (intra-trial) and experience distillation across multiple attempts (cross-trial). We trace these challenges to an empirical informational asymmetry in GUI navigation: while expected transitions can often be compressed into lightweight textual summaries, unexpected outcomes benefit from preserved screenshots as causal evidence for accurate diagnosis. Building on this insight, we propose AnchorGUI, a unified framework driven by the Cognitive State Anchor (CSA). The CSA acts as a per-step primitive that actively compares expected and observed transitions, converting passive multimodal trajectories into explicit prediction-error signals. These signals orchestrate a dual-scale learning mechanism via an asymmetric memory. For intra-trial correction, a sliding window selectively retains visual evidence for detected mismatches, providing immediate, visually-grounded feedback. For cross-trial distillation, this asymmetric memory focuses the computationally expensive credit assignment search space on likely failure steps. Experiments across four benchmarks validate the effectiveness of our approach. On AndroidWorld, AnchorGUI achieves a 57.3% success rate with a $2.4\times$ token reduction per step. Furthermore, cross-trial distillation reaches 69.2% success (+11.9% gain), significantly outperforming standard reflection methods while maintaining sub-linear context scaling.

Problem

• GUI Navigation
• Vision-Language Models
• Visual Histories
• Intra-Trial
• Cross-Trial

Innovation

• Asymmetric Memory
• Dual-Scale Learning
• Cognitive State Anchor (CSA)
• Prediction-Error Signals
• GUI Navigation