APeB: Benchmarking Personalization Ability of Large Language Model Agents

2026-07-03

AI Summary

This study addresses the challenge of personalizing large language model (LLM) agents when handling users’ raw, ambiguous queries—a task hindered by difficulties in intent inference, preference extraction, and multi-option decision-making. To this end, the authors introduce the first personalized product search (PPS) evaluation platform grounded in real user behavior, along with the Agent Personalization Benchmark (APeB), and propose a scoring-rule-based evaluation methodology. They further develop history-aware query refinement techniques, notably the Variational Query Refinement Agent (VQRA). Experimental results demonstrate that while mainstream LLMs perform well on explicit queries, their effectiveness is limited during early, ambiguous interaction stages. In contrast, a multi-step agent workflow integrating VQRA substantially improves performance, underscoring the critical role of dedicated modules that leverage historical interaction data for personalization.

Abstract

LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives. Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories. We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from action logs, pairing underspecified intents with rich histories and user-viewed candidate items. Evaluating state-of-the-art LLMs with multi-step agent workflows, we find that models handle explicit queries well but struggle with early-stage queries requiring intent and preference discovery. Rubric analysis attributes this gap mainly to ineffective history use. A simple history-aware query-refinement pipeline, VQRA, yields consistent gains, highlighting the need for dedicated history-utilization modules in personalized agents.

Problem

• personalization
• large language models
• underspecified queries
• user intent
• interaction history

Innovation

• personalization
• large language model agents
• benchmark
• query refinement
• interaction history