ClawGym II: Exploring Black-Box RL on Agent Harness

2026-08-17

AI Summary

This study addresses the challenges of poor scalability and training instability in reinforcement learning for complex agentic tool use by proposing a unified black-box RL framework. By integrating sandbox isolation, prefix-tree-based trajectory reconstruction, and a hybrid tool training mechanism, the approach decouples policy optimization from heterogeneous system dynamics while remaining compatible with PPO and GRPO algorithms. Experimental results demonstrate consistent performance gains across multiple complex tasks, achieving a maximum improvement of 14.81 percentage points in pass@1 accuracy. These findings confirm that the proposed method effectively enables stable and scalable optimization of general-purpose agents operating within intricate tool-use environments, thereby overcoming critical bottlenecks in current agentic RL training paradigms.

Abstract

Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.

Problem

• Black-Box RL
• Agent Harness
• Long-horizon Tasks
• Scalable Optimization

Innovation

• Black-Box RL
• Agent Harness
• Prefix Tree Trajectory Reconstruction
• Mix-Harness Training
• Sandbox Infrastructure