EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents

2026-07-07

AI Summary

This work addresses the challenge of learning long-horizon tasks in multi-turn computer usage scenarios by overcoming the limitations of static imitation learning, which fails to model the closed-loop dynamics of real interactive feedback. The paper introduces the first agent framework that supports online reinforcement learning, enabling continuous policy refinement through interaction with an executable sandbox environment and optimization based on verifiable task outcomes. Key innovations include Step-Level Policy Optimization (STEPO) to maintain trajectory advantage balance, a Dynamic Tri-Adaptive Curriculum (DTAC), policy-aware filtering, an asynchronous RL architecture, and mini-batch grouping, collectively ensuring stable and efficient multi-turn online learning. Evaluated on the OSWorld-Verified benchmark, the method achieves a 63.2% success rate, substantially outperforming open-source baselines of comparable scale and approaching the performance of significantly larger models.

Abstract

Computer-use agents must solve long-horizon tasks through repeated interaction with partially observable, multimodal desktop environments. Although imitation learning and offline trajectory refinement provide strong priors, static traces cannot cover the causal feedback loop of real computer use: each action changes the screen state, future action space, and recovery options. EvoCUA-1.5 extends self-evolving computer-use agents from offline experience learning to online reinforcement learning, where policies interact with executable sandbox environments and improve from verifiable task outcomes. Online RL in this setting requires more than directly reusing single-turn language-RL recipes. Multi-turn interaction introduces context-managed observations, sparse terminal rewards, variable-length trajectories, and slow environment feedback. EvoCUA-1.5 addresses these challenges with Step-Level Policy Optimization (STEPO), which preserves trajectory-level advantage balance after decomposition into step-level samples; policy-aware filtering and pass-rate calibration over verifiable synthesized tasks; Dynamic Tri-Adaptive Curriculum (DTAC), which combines learnable tasks, difficult positive replay, and controlled infeasible-task exposure; and a fully asynchronous RL infrastructure with staleness control and mini-group batching. Experiments show that these components improve training stability and downstream performance. EvoCUA-1.5 achieves 63.2\% success on OSWorld-Verified, outperforming comparable 32B/35B-scale open-weight baselines and even approaching models with significantly larger parameter counts. Overall, EvoCUA-1.5 provides a practical framework for scaling online RL in multi-turn computer-use agents.

Problem

• multi-turn computer-use agents
• online reinforcement learning
• long-horizon tasks
• partially observable environments
• sparse rewards

Innovation

• Online Reinforcement Learning
• Multi-turn Computer-Use Agents
• Step-Level Policy Optimization
• Dynamic Tri-Adaptive Curriculum
• Asynchronous RL Infrastructure