Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

2026-02-18

AI Summary

This work addresses the challenge that existing web-based agents struggle to perform personalized reasoning when user intent is ambiguous, particularly in leveraging historical preferences. To this end, we propose the “Clarify for Personalization” principle and introduce the first benchmark tailored for evaluating personalized agents in open-web environments, which implicitly models users’ long-term history to interpret ambiguous queries. We design an evaluation framework that integrates long-term user history with ambiguous queries and incorporate a reasoning-aware, fine-grained assessment mechanism. Comprehensive experiments across diverse agent architectures, language model backbones, history utilization strategies, and levels of query ambiguity reveal critical bottlenecks in current approaches regarding contextual reasoning. The codebase and dataset are publicly released to foster reproducible research in this emerging area.

Technology Category

• Knowledge Representation and Reasoning: Preferences
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Large language models have advanced web agents, yet current agents lack personalization capabilities. Since users rarely specify every detail of their intent, practical web agents must be able to interpret ambiguous queries by inferring user preferences and contexts. To address this challenge, we present Persona2Web, the first benchmark for evaluating personalized web agents on the real open web, built upon the clarify-to-personalize principle, which requires agents to resolve ambiguity based on user history rather than relying on explicit instructions. Persona2Web consists of: (1) user histories that reveal preferences implicitly over long time spans, (2) ambiguous queries that require agents to infer implicit user preferences, and (3) a reasoning-aware evaluation framework that enables fine-grained assessment of personalization. We conduct extensive experiments across various agent architectures, backbone models, history access schemes, and queries with varying ambiguity levels, revealing key challenges in personalized web agent behavior. For reproducibility, our codes and datasets are publicly available at https://anonymous.4open.science/r/Persona2Web-73E8.

Problem

• personalized web agents
• contextual reasoning
• user history
• ambiguous queries
• preference inference

Innovation

• personalized web agents
• contextual reasoning
• user history
• ambiguous query resolution
• benchmarking