🤖 AI Summary
Existing GUI agents face two critical challenges: unreliable reward signals and limited capacity for online trajectory generation—leading to unreliable reasoning and low data efficiency. To address these, we propose Orcust, a novel framework introducing two key mechanisms: (1) Principle-Constrained Reward Modeling (PCRM), which incorporates interpretable, human-aligned rules into the reinforcement learning reward function to ensure policy adherence to domain principles; and (2) Online Virtual Machine-Grounded Trajectory Construction (OVTC), which leverages a lightweight virtual machine to autonomously generate high-quality, structured interaction trajectories enabling fine-grained, stepwise training. Integrating chain-of-thought reasoning with LLM-driven rule feedback, Orcust achieves +22.2% and +23.9% improvements on ScreenSpot and ScreenSpot-Pro benchmarks, respectively. The framework significantly enhances reasoning reliability, task adaptability, and system scalability.
📝 Abstract
Recent advances in GUI agents have achieved remarkable grounding and action-prediction performance, yet existing models struggle with unreliable reward signals and limited online trajectory generation. In this paper, we introduce Orcust, a framework that integrates Principle-Constrained Reward Modeling (PCRM) and Online VM-Grounded Trajectory Construction (OVTC) to enhance reasoning reliability and data efficiency in interactive GUI tasks. We leverages environment-verifiable and LLM-derived principle to enforce interpretable reward signals that constrain long chain-of-thought reasoning and rule-based feedback. OVTC spins up instrumented virtual machines to autonomously collect structured GUI interaction trajectories with explicit procedural and structural objectives, enabling the training of a stepwise reward model that robustly captures human preferences and adheres to task-specific constraints. Extensive experiments on standard GUI benchmarks covering perceptual grounding, foundational operations, and end-to-end task execution reveal that Orcust achieves state-of-the-art performance, improving by 22.2% on ScreenSpot and 23.9% on ScreenSpot-Pro over the base model (i.e. Qwen2.5-VL-7B). The results demonstrate Orcust's effectiveness in enhancing the reasoning, adaptability and scalability of GUI agents across various environments and task complexities.