A Task-State Representation for Long-Horizon Mobile GUI Agents

2026-07-01

AI Summary

Existing long-horizon mobile GUI agents struggle to distinguish persistent task states from transient screen observations, often leading to goal forgetting or hallucination. This work proposes a Task State Representation (TSR) framework that explicitly decouples task state from perceptual input through a lightweight external wrapper, without modifying the underlying model architecture or requiring fine-tuning. TSR integrates a global instruction summary, a dynamic subgoal tracker, and an action validator, continuously updating the task state via visual comparisons before and after each action. This approach achieves, for the first time, training-agnostic separation of task state and observation, significantly boosting performance across four mobile GUI benchmarks—improving success rates by up to 12 percentage points on complex cross-app and memory-intensive tasks.

Abstract

While long-horizon mobile GUI agents typically rely on thought-action-observation loops, they struggle to separate persistent task states from transient screen observations. As execution histories grow, this entanglement imposes a severe context burden, causing agents to forget initial requirements, hallucinate progress, or repeatedly interact with stale interfaces. To address this, we introduce Task-State Representation (TSR), a training-free framework that explicitly decouples task state from sensory input. Acting as a lightweight external wrapper, TSR maintains three structured components: a global instruction summary, a dynamic progress tracker for subgoals, and a transition-aware action verifier. By continuously updating through pre- and post-action visual comparisons, TSR effectively guides the agent's reasoning without requiring architectural modifications. Experiments across four mobile GUI benchmarks validate TSR's effectiveness, yielding up to a 12 absolute point increase in success rate on complex cross-application and memory-intensive tasks.

Problem

• long-horizon
• mobile GUI agents
• task state
• screen observations
• context burden

Innovation

• Task-State Representation
• mobile GUI agents
• long-horizon reasoning
• state decoupling
• training-free framework