GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models
🤖 AI Summary
This work addresses the challenge of task failure in GUI agents caused by irreversible erroneous actions by proposing the first test-time expansion critique mechanism (TTS) tailored for GUI agents. The approach establishes a data flywheel system that employs an Intuition Critique Model (ICM) to dynamically evaluate and filter agent actions in real time, while simultaneously guiding the mining of high-quality positive and negative samples. This enables co-evolutionary self-improvement between the ICM and the agent. Leveraging large vision-language models and a data flywheel training strategy, the ICM significantly enhances the test-time performance of both open- and closed-source GUI agents across multiple benchmarks, with continual gains as data is iteratively reused—thereby transcending conventional static evaluation paradigms.
Application Category
📝 Abstract
While Large Vision-Language Models (LVLMs) have significantly advanced GUI agents'capabilities in parsing textual instructions, interpreting screen content, and executing tasks, a critical challenge persists: the irreversibility of agent operations, where a single erroneous action can trigger catastrophic deviations. To address this, we propose the GUI Action Critic's Data Flywheel System (GAIA), a training framework that enables the models to have iterative critic capabilities, which are used to improve the Test-Time Scaling (TTS) of basic GUI agents'performance. Specifically, we train an Intuitive Critic Model (ICM) using positive and negative action examples from a base agent first. This critic evaluates the immediate correctness of the agent's intended actions, thereby selecting operations with higher success probability. Then, the initial critic guides agent actions to collect refined positive/negative samples, initiating the self-improving cycle. The augmented data then trains a second-round critic with enhanced discernment capability. We conduct experiments on various datasets and demonstrate that the proposed ICM can improve the test-time performance of various closed-source and open-source models, and the performance can be gradually improved as the data is recycled. The code and dataset will be publicly released.
Problem
Research questions and friction points this paper is trying to address.
GUI agents
irreversible actions
test-time scaling
action criticism
catastrophic deviations
Innovation
Methods, ideas, or system contributions that make the work stand out.
Data Flywheel
Test-Time Scaling
GUI Agent
Critic Model
Iterative Self-Improvement
🔎 Similar Papers
No similar papers found.
