CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents

2026-05-25

AI Summary

This work addresses the scarcity of scalable, high-quality training data with reliable and deterministic rewards for computer-using agents (CUAs), as existing benchmarks either offer limited coverage or suffer from inconsistent reward signals. To overcome this, the authors propose CUA-Gym, a novel multi-agent framework that collaboratively generates triplets of task instructions, environment states, and verifiable reward functions through a pipeline integrating LLM majority voting, agent rollback validation, and adversarial iterative refinement. This approach yields CUA-Gym-Hub, a high-fidelity collection of web simulation environments comprising 32,112 validated samples across 110 distinct settings. Models trained on this dataset—CUA-Gym-A3B and CUA-Gym-A17B—achieve 62.1% and 72.6% success rates on OSWorld-Verified, substantially outperforming open-source counterparts of comparable scale, and demonstrate strong generalization capabilities on WebArena.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven breakthroughs in domains such as math, tool-use, and software engineering, yet its extension to computer-use agents (CUAs) has been bottlenecked by the scarcity of scalable training data with deterministic rewards. Constructing such data for CUAs requires consistent task instruction, executable environment, and verifiable reward. However, hand-curated benchmarks achieve high reward fidelity but cover few applications and LLM-as-judge-based datasets scale broadly but lack reliable verification. We present CUA-Gym, a scalable pipeline that co-generates task instructions, environment states, and reward functions. Concretely, a Generator agent constructs the initial and golden environment states, and a separate Discriminator agent writes the reward function from the task specification. An orchestrator agent drives the two through iterative rounds upon execution. Generated tuples then pass a final filter combining LLM majority voting and agent rollouts, ensuring quality beyond the per-task adversarial loop. To address the scarcity of training environments, we further synthesize CUA-Gym-Hub, a broad suite of high-fidelity mock web applications grounded in real-world software-use distributions, expanding the scale of CUA RLVR data by magnitude. Using this pipeline, we construct CUA-Gym, a dataset of 32,112 verified RLVR training tuples grounded in 110 environments. Trained with GSPO on CUA-Gym, our CUA-Gym-A3B and CUA-Gym-A17B achieve 62.1% and 72.6% on OSWorld-Verified, outperforming prior open-source CUAs at comparable scales, with performance scaling smoothly in both data volume and environment diversity. The same checkpoints also improve on the held-out WebArena benchmark, indicating transfer beyond the training environments. We will open-source the full synthesis pipeline, dataset, CUA-Gym-Hub environments, and models.

Problem

• computer-use agents
• verifiable rewards
• scalable training data
• reinforcement learning
• reward verification

Innovation

• Reinforcement Learning with Verifiable Rewards
• Computer-Use Agents
• Synthetic Environment Generation
• Adversarial Agent Collaboration
• Scalable RLVR Dataset