PRO-CUA: Process-Reward Optimization for Computer Use Agents

2026-05-27

AI Summary

This work addresses key challenges in training intelligent agents, including high costs of online interaction, sparse rewards, difficult credit assignment, and distributional shift in behavioral cloning. To overcome these issues, the authors propose a process reward optimization framework that decouples policy interaction from optimization and introduces a step-level process reward model. This model provides fine-grained feedback without requiring expert demonstrations or ground-truth answers, enabling dense credit assignment. By integrating group-relative advantage estimation, online rolling sampling, and diverse action generation, the method significantly enhances both performance and training stability of agents on long-horizon GUI tasks, as demonstrated on real-world web benchmarks.

Abstract

Computer use agents (CUAs) have shown strong potential for automating complex digital workflows, yet their training remains constrained by costly live environment interaction and limited high-quality supervision. Existing filtered behavior cloning pipelines suffer from imitation bottlenecks, including distribution shift from the expert demonstration and the absence of negative learning signals. Meanwhile, standard trajectory-level reinforcement learning struggles with sparse rewards, ambiguous credit assignment, and high infrastructure costs for long-horizon GUI interaction. In this work, we propose PRO-CUA, a process-reward optimization framework for training CUAs with iterative step-level reinforcement learning. PRO-CUA decouples on-policy environment interaction from policy optimization: the current policy collects states through live rollouts, generates diverse candidate actions for each state, receives step-level feedback from a process reward model (PRM), and is optimized with group-relative advantages. This design enables dense and flexible credit assignment without relying on golden answers or offline expert trajectories, while reducing distribution shift by training on the agent's own execution states. Experiments on live web benchmarks demonstrate the effectiveness of PRO-CUA and the reliability of PRM-guided step-level training.

Problem

• computer use agents
• behavior cloning
• reinforcement learning
• distribution shift
• sparse rewards

Innovation

• process-reward optimization
• step-level reinforcement learning
• computer use agents
• group-relative advantages
• distribution shift mitigation