ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration

2025-09-25

AI Summary

GUI agents face dual challenges: sparse and hard-to-obtain reward signals, and poor generalization and low accuracy of existing reward estimation methods (e.g., rule-based, model-based, or LLM-as-a-judge approaches). To address these, we propose an *active reward system* featuring a novel collaborative mechanism between a *reasoner* and an *evaluation executor*. The reasoner dynamically schedules tasks and triggers interface-state probing; the evaluation executor interacts with the environment to acquire real-time observations, constructing verifiable, dynamic evidence chains—thereby overcoming reliance on static judgments or ground-truth trajectories. Our method integrates LLM-driven task scheduling, domain-specific evaluation agents, active probing, and closed-loop feedback. Evaluated on 3,000+ interaction trajectories, it achieves +5.3% absolute improvement in reward accuracy and +19.4% in F1 score. When integrated with state-of-the-art policy agents, task success rate increases by 22.4%.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Reward is critical to the evaluation and training of large language models (LLMs). However, existing rule-based or model-based reward methods struggle to generalize to GUI agents, where access to ground-truth trajectories or application databases is often unavailable, and static trajectory-based LLM-as-a-Judge approaches suffer from limited accuracy. To address these challenges, we propose ProRe, a proactive reward system that leverages a general-purpose reasoner and domain-specific evaluator agents (actors). The reasoner schedules targeted state probing tasks, which the evaluator agents then execute by actively interacting with the environment to collect additional observations. This enables the reasoner to assign more accurate and verifiable rewards to GUI agents. Empirical results on over 3K trajectories demonstrate that ProRe improves reward accuracy and F1 score by up to 5.3% and 19.4%, respectively. Furthermore, integrating ProRe with state-of-the-art policy agents yields a success rate improvement of up to 22.4%.

Problem

• Addresses inaccurate reward systems for GUI agents
• Overcomes limitations of static trajectory-based evaluation methods
• Enables verifiable rewards through proactive environment interaction

Innovation

• Proactive reward system with reasoner-actor collaboration
• Reasoner schedules targeted state probing tasks
• Evaluator agents collect observations through environment interaction