WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks

2026-01-05

arXiv.org

AI Summary

Existing training environments for visual web agents predominantly rely on synthetic or small-scale tasks, which are insufficient for learning robust policies in real-world, non-stationary, and diverse websites. To address this limitation, this work introduces WebGym, the largest open-source training environment to date, comprising nearly 300,000 human-annotated, reward-defined tasks drawn from real websites, with the novel inclusion of hierarchical task difficulty levels. We propose a reinforcement learning framework based on agent interaction trajectories, integrating a high-throughput asynchronous rollout system with fine-tuning of the Qwen-3-VL-8B-Instruct vision-language model. Evaluated on an out-of-domain test set of entirely unseen websites, our approach achieves a success rate of 42.9%, substantially outperforming GPT-4o (27.1%), GPT-5-Thinking (29.8%), and existing baselines (26.2%).

Technology Category

• Computer Vision: Large Vision Models
• Multiagent Systems: Adversarial Agents
• Machine Learning: Reinforcement Learning

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present WebGym, the largest-to-date open-source environment for training realistic visual web agents. Real websites are non-stationary and diverse, making artificial or small-scale task sets insufficient for robust policy learning. WebGym contains nearly 300,000 tasks with rubric-based evaluations across diverse, real-world websites and difficulty levels. We train agents with a simple reinforcement learning (RL) recipe, which trains on the agent's own interaction traces (rollouts), using task rewards as feedback to guide learning. To enable scaling RL, we speed up sampling of trajectories in WebGym by developing a high-throughput asynchronous rollout system, designed specifically for web agents. Our system achieves a 4-5x rollout speedup compared to naive implementations. Second, we scale the task set breadth, depth, and size, which results in continued performance improvement. Fine-tuning a strong base vision-language model, Qwen-3-VL-8B-Instruct, on WebGym results in an improvement in success rate on an out-of-distribution test set from 26.2% to 42.9%, significantly outperforming agents based on proprietary models such as GPT-4o and GPT-5-Thinking that achieve 27.1% and 29.8%, respectively. This improvement is substantial because our test set consists only of tasks on websites never seen during training, unlike many other prior works on training visual web agents.

Problem

• visual web agents
• realistic tasks
• non-stationary environments
• task diversity
• robust policy learning

Innovation

• WebGym
• visual web agents
• asynchronous rollout
• reinforcement learning
• vision-language model