Rethinking Token Pruning for Historical Screenshots in GUI Visual Agents: Semantic, Spatial, and Temporal Perspectives

2026-03-26

AI Summary

This work addresses the excessive computational overhead incurred by high-resolution GUI screenshots in visual agents, which introduce a large number of visual tokens that hinder efficient retention of historical information. The authors systematically investigate token pruning strategies across semantic, spatial, and temporal dimensions: leveraging edge detection to separate foreground from background and revealing the auxiliary role of background regions in state transitions; identifying the advantage of random pruning in preserving spatial structure; and proposing a recency-biased dynamic token allocation mechanism. Experimental results demonstrate that the proposed approach substantially reduces computational cost while maintaining near-optimal task performance, offering practical and principled design guidelines for efficient GUI-based visual agents.

Abstract

In recent years, GUI visual agents built upon Multimodal Large Language Models (MLLMs) have demonstrated strong potential in navigation tasks. However, high-resolution GUI screenshots produce a large number of visual tokens, making the direct preservation of complete historical information computationally expensive. In this paper, we conduct an empirical study on token pruning for historical screenshots in GUI scenarios and distill three practical insights that are crucial for designing effective pruning strategies. First, we observe that GUI screenshots exhibit a distinctive foreground-background semantic composition. To probe this property, we apply a simple edge-based separation to partition screenshots into foreground and background regions. Surprisingly, we find that, contrary to the common assumption that background areas have little semantic value, they effectively capture interface-state transitions, thereby providing auxiliary cues for GUI reasoning. Second, compared with carefully designed pruning strategies, random pruning possesses an inherent advantage in preserving spatial structure, enabling better performance under the same computational budget. Finally, we observe that GUI Agents exhibit a recency effect similar to human cognition: by allocating larger token budgets to more recent screenshots and heavily compressing distant ones, we can significantly reduce computational cost while maintaining nearly unchanged performance. These findings offer new insights and practical guidance for the design of efficient GUI visual agents.

Problem

• token pruning
• GUI visual agents
• historical screenshots
• computational efficiency
• visual tokens

Innovation

• token pruning
• GUI visual agents
• semantic composition
• spatial structure preservation
• recency effect