WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model

2025-04-23

AI Summary

Existing web agents often suffer from performance stagnation due to insufficient environmental exploration and underutilization of pre-trained web knowledge. To address this, we propose the Collaboratively Evolved World Model LLM—a novel framework that jointly optimizes a world model and a web agent for the first time. Our approach leverages the LLM’s internalized web knowledge to construct a world model capable of both synthetic data generation and forward-looking simulation. It enhances policy optimization and decision robustness via self-supervised trajectory generation, forward observation prediction, and multi-step action-guided simulation during reasoning. Evaluated on three real-world web benchmarks—Mind2Web-Live, WebVoyager, and GAIA-web—our method achieves an average 10% improvement in task completion rate. Crucially, it requires no distillation from proprietary large language models, demonstrating strong continuous adaptive learning capability.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Agent self-improvement, where the backbone Large Language Model (LLM) of the agent are trained on trajectories sampled autonomously based on their own policies, has emerged as a promising approach for enhancing performance. Recent advancements, particularly in web environments, face a critical limitation: their performance will reach a stagnation point during autonomous learning cycles, hindering further improvement. We argue that this stems from limited exploration of the web environment and insufficient exploitation of pre-trained web knowledge in LLMs. To improve the performance of self-improvement, we propose a novel framework that introduces a co-evolving World Model LLM. This world model predicts the next observation based on the current observation and action within the web environment. Leveraging LLMs' pretrained knowledge of abundant web content, the World Model serves dual roles: (1) as a virtual web server generating self-instructed training data to continuously refine the agent's policy, and (2) as an imagination engine during inference, enabling look-ahead simulation to guide action selection for the agent LLM. Experiments in real-world web environments (Mind2Web-Live, WebVoyager, and GAIA-web) show a 10% performance gain over existing self-evolving agents, demonstrating the efficacy and generalizability of our approach, without using any distillation from more powerful close-sourced models. Our work establishes the necessity of integrating world models into autonomous agent frameworks to unlock sustained adaptability.

Problem

• Overcoming stagnation in web agent self-improvement cycles
• Enhancing exploration and pre-trained web knowledge utilization
• Integrating co-evolving world models for sustained adaptability

Innovation

• Introduces co-evolving World Model LLM
• Generates self-instructed training data
• Enables look-ahead simulation for action