Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

2026-06-23

AI Summary

This work addresses the challenge of sparse reward signals in reinforcement learning for graphical user interface (GUI) tasks, where success hinges on visual judgment and is difficult to capture via handcrafted rules or human annotations. The authors propose a reinforcement learning fine-tuning framework that leverages autonomous vision-language evaluation, using the alignment score between the final screenshot and the task instruction—generated by a vision-language model—as a terminal reward without requiring task-specific rules or human labels. To mitigate the inherent noise in this reward signal, they introduce a noise-corrected reward estimator integrated with the Proximal Policy Optimization (PPO) algorithm for unsupervised reward generation and policy optimization. Experiments demonstrate that the method improves average success rates by 12.6 percentage points over zero-shot baselines and by 5.1 percentage points over uncorrected reward variants across macOSWorld, Windows Agent Arena, and OSWorld benchmarks.

Abstract

Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces. However, reinforcement learning for CUAs remains difficult because open-ended desktop environments rarely provide scalable, machine-readable reward signals: task success is often visually grounded and hard to specify with handcrafted reward functions or dense manual labels. We propose an RL fine-tuning framework that uses autonomous vision-language evaluation as a scalable supervision signal for GUI agents. Given a final screenshot and the original instruction, a Vision-Language Model judges task completion and provides terminal feedback without task-specific heuristics or manual labels during policy optimization. Because autonomous evaluators are imperfect, we model their feedback as a noisy binary reward channel and derive a noise-corrected reward estimator for Proximal Policy Optimization. Experiments across macOSWorld, Windows Agent Arena, and OSWorld show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning. These results suggest that autonomous evaluation can serve as a practical reward signal for RL in GUI environments when evaluator noise is explicitly modeled and corrected.

Problem

• Reinforcement Learning
• Computer-Use Agents
• Reward Signal
• Graphical User Interfaces
• Autonomous Evaluation

Innovation

• Reinforcement Learning
• Vision-Language Model
• Autonomous Evaluation
• Noise-Corrected Reward
• GUI Agents