WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representation

2025-10-21

AI Summary

Existing web agent evaluation methods predominantly rely on binary success metrics or single-reference trajectory matching, overlooking the inherent path diversity and behavioral structure within benchmarks. To address this, we propose WebGraphEval—the first framework to model multi-turn interaction trajectories as weighted action graphs. Our approach standardizes actions, merges trajectories, and leverages graph-structured modeling to enable unified, cross-agent, multi-path, and efficiency-aware evaluation. It supports reward propagation and success-weighted edge statistics, enabling precise localization of critical decision points, identification of redundant operations, and quantification of behavioral discrepancies across agents. We validate WebGraphEval on benchmarks including WebArena, analyzing thousands of trajectories from six distinct agent families. Results demonstrate substantial improvements in evaluation granularity, interpretability, and insight into agent behavior—advancing both diagnostic capability and principled benchmarking for web navigation agents.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Current evaluation of web agents largely reduces to binary success metrics or conformity to a single reference trajectory, ignoring the structural diversity present in benchmark datasets. We present WebGraphEval, a framework that abstracts trajectories from multiple agents into a unified, weighted action graph. This representation is directly compatible with benchmarks such as WebArena, leveraging leaderboard runs and newly collected trajectories without modifying environments. The framework canonically encodes actions, merges recurring behaviors, and applies structural analyses including reward propagation and success-weighted edge statistics. Evaluations across thousands of trajectories from six web agents show that the graph abstraction captures cross-model regularities, highlights redundancy and inefficiency, and identifies critical decision points overlooked by outcome-based metrics. By framing web interaction as graph-structured data, WebGraphEval establishes a general methodology for multi-path, cross-agent, and efficiency-aware evaluation of web agents.

Problem

• Evaluates web agents using graph-based multi-turn trajectory analysis
• Addresses limitations of binary success metrics in web agent evaluation
• Identifies critical decision points and inefficiencies across multiple agents

Innovation

• Abstracts agent trajectories into unified weighted action graph
• Applies structural analyses like reward propagation and edge statistics
• Establishes graph-based methodology for multi-path cross-agent evaluation