Discriminative World Models for Web Agents

2026-09-02

AI Summary

为提高网页代理决策准确性，本文提出预测状态匹配方法训练世界模型，使其能更好区分不同动作导致的状态，从而优化行动选择。

Abstract

Recent web agents use world models for test-time action selection by sampling candidate actions, predicting the resulting web states, and ranking them with a ranker model or a Process Reward Model (PRM). These world models are typically trained via supervised next-state prediction to generate fixed representations like HTML or AXTree snapshots. However, this objective is misaligned with the downstream ranker, which relies on predicted states being discriminative across candidates to accurately score them. To address this, we introduce predicted-state matching, a training objective where the predicted representation must distinguish the true resulting state from those reached by alternative actions. We train these models using a branching web-agent dataset derived from WebArena Go-Browse trajectories, where every decision point contains multiple alternative actions and their resulting states. Experiments on our held-out predicted-state matching benchmark show that our approach outperforms world models trained with supervised next-state prediction. We further show that our approach improves PRM-style action ranking on WebPRMBench compared with action-only PRMs and PRMs augmented with supervised-next-state world models. Finally, on WebArena-Lite, using our world model for test-time action selection improves end-to-end task success. Our project page is available at: https://dhruvpendharkar.github.io/dwm/.

Problem

• web agents
• world models
• next-state prediction
• ranker model
• Process Reward Model (PRM)

Innovation

• predicted-state matching
• world models
• discriminative states
• web agents
• Process Reward Model (PRM)