Benchmarking and Improving GUI Agents in High-Dynamic Environments

2026-04-28

AI Summary

This work addresses the limitations of existing GUI agents that rely solely on single-frame screenshots, which struggle to perceive dynamic interface states and make effective decisions in highly interactive environments. The study presents the first systematic investigation of this challenge, introducing DynamicUI—an agent that takes screen recordings as input—and DynamicGUIBench, a new evaluation benchmark. DynamicUI extracts dynamic context through video frame clustering and caption generation, and enhances reasoning consistency and decision-making via action-conditioned refinement and trajectory reflection mechanisms. Experimental results demonstrate that DynamicUI significantly outperforms current methods on DynamicGUIBench while maintaining competitive performance on other established public benchmarks.

Abstract

Recent advancements in Graphical User Interface (GUI) agents have predominantly focused on training paradigms like supervised fine-tuning (SFT) and reinforcement learning (RL). However, the challenge of high-dynamic GUI environments remains largely underexplored. Existing agents typically rely on a single screenshot after each action for decision-making, leading to a partially observable (or even unobservable) Markov decision process, where the key GUI state including important information for actions is often inadequately captured. To systematically explore this challenge, we introduce DynamicGUIBench, a comprehensive online GUI benchmark spanning ten applications and diverse interaction scenarios characterized by important interface changes between actions. Furthermore, we present DynamicUI, an agent designed for dynamic interfaces, which takes screen-recording videos of the interaction process as input and consists of three components: a dynamic perceiver, a refinement strategy, and a reflection. Specifically, the dynamic perceiver clusters frames of the GUI video, generates captions for the centroids, and iteratively selects the most informative frames as the salient dynamic context. Considering that there may be inconsistencies and noise between the selected frames and the textual context of the agent, the refinement strategy employs an action-conditioned filtering to refine thoughts to mitigate thought-action inconsistency and redundancy. Based on the refined agent trajectories, the reflection module provides effective and accurate guidance for further actions. Experiments on DynamicGUIBench demonstrate that DynamicUI significantly improves the performance in dynamic GUI environments, while maintaining competitive performance on other public benchmarks.

Problem

• GUI agents
• high-dynamic environments
• partially observable MDP
• interface changes
• state observation

Innovation

• DynamicGUIBench
• DynamicUI
• screen-recording video
• dynamic perceiver
• action-conditioned filtering