BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism

2025-05-27

AI Summary

Existing GUI agents prioritize single-step action accuracy but lack robust error detection and recovery capabilities, resulting in low success rates for multi-step tasks. To address this, we propose a novel Verifier-Judger-Reflector (VJR) collaborative backtracking mechanism that, for the first time, integrates action verification, execution outcome judgment, and behavioral reflection within a unified reinforcement learning framework. We further design a dedicated training dataset focused on result pages and a judgment-aware reward function to guide effective policy learning. Our approach significantly enhances agent fault tolerance and self-correction capability in complex GUI interactions. On the Mobile3M and Auto-UI benchmarks, it improves task success rates by 12.7% and 9.4%, respectively, and boosts step-level accuracy by 8.2–15.6%. The code and dataset will be publicly released.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Graphical User Interface (GUI) agents have gained substantial attention due to their impressive capabilities to complete tasks through multiple interactions within GUI environments. However, existing agents primarily focus on enhancing the accuracy of individual actions and often lack effective mechanisms for detecting and recovering from errors. To address these shortcomings, we propose the BacktrackAgent, a robust framework that incorporates a backtracking mechanism to improve task completion efficiency. BacktrackAgent includes verifier, judger, and reflector components as modules for error detection and recovery, while also applying judgment rewards to further enhance the agent's performance. Additionally, we develop a training dataset specifically designed for the backtracking mechanism, which considers the outcome pages after action executions. Experimental results show that BacktrackAgent has achieved performance improvements in both task success rate and step accuracy on Mobile3M and Auto-UI benchmarks. Our data and code will be released upon acceptance.

Problem

• Enhancing GUI agent error detection and recovery
• Improving task completion efficiency with backtracking
• Developing training data for backtracking mechanism outcomes

Innovation

• BacktrackAgent framework with error recovery
• Verifier, judger, reflector for error detection
• Training dataset for backtracking mechanism