DynaWeb: Model-Based Reinforcement Learning of Web Agents

2026-01-29

AI Summary

Training autonomous web agents faces significant challenges due to the low efficiency, high cost, and substantial risks associated with real-world internet interactions. To address this, this work proposes DynaWeb, a novel framework that introduces model-based reinforcement learning (MBRL) to web agents for the first time. DynaWeb constructs a world model capable of predicting web states, enabling efficient generation of interaction trajectories in a synthetic environment, which are then combined with expert demonstrations for hybrid training. This approach allows the agent to “imagine” future interactions, substantially improving both sample efficiency and policy performance. Evaluated on the WebArena and WebVoyager benchmarks, DynaWeb significantly outperforms existing open-source web agents, demonstrating its effectiveness and scalability.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Sustainability of Web economics

Abstract

The development of autonomous web agents, powered by Large Language Models (LLMs) and reinforcement learning (RL), represents a significant step towards general-purpose AI assistants. However, training these agents is severely hampered by the challenges of interacting with the live internet, which is inefficient, costly, and fraught with risks. Model-based reinforcement learning (MBRL) offers a promising solution by learning a world model of the environment to enable simulated interaction. This paper introduces DynaWeb, a novel MBRL framework that trains web agents through interacting with a web world model trained to predict naturalistic web page representations given agent actions. This model serves as a synthetic web environment where an agent policy can dream by generating vast quantities of rollout action trajectories for efficient online reinforcement learning. Beyond free policy rollouts, DynaWeb incorporates real expert trajectories from training data, which are randomly interleaved with on-policy rollouts during training to improve stability and sample efficiency. Experiments conducted on the challenging WebArena and WebVoyager benchmarks demonstrate that DynaWeb consistently and significantly improves the performance of state-of-the-art open-source web agent models. Our findings establish the viability of training web agents through imagination, offering a scalable and efficient way to scale up online agentic RL.

Problem

• web agents
• reinforcement learning
• model-based reinforcement learning
• LLM
• online RL

Innovation

• Model-Based Reinforcement Learning
• Web Agents
• World Model
• Imagined Rollouts
• Sample Efficiency