Problem
Research questions and friction points this paper is trying to address.
personalization
large language models
underspecified queries
user intent
interaction history
APeB: Benchmarking Personalization Ability of Large Language Model Agents
🤖 AI Summary
This study addresses the challenge of personalizing large language model (LLM) agents when handling users’ raw, ambiguous queries—a task hindered by difficulties in intent inference, preference extraction, and multi-option decision-making. To this end, the authors introduce the first personalized product search (PPS) evaluation platform grounded in real user behavior, along with the Agent Personalization Benchmark (APeB), and propose a scoring-rule-based evaluation methodology. They further develop history-aware query refinement techniques, notably the Variational Query Refinement Agent (VQRA). Experimental results demonstrate that while mainstream LLMs perform well on explicit queries, their effectiveness is limited during early, ambiguous interaction stages. In contrast, a multi-step agent workflow integrating VQRA substantially improves performance, underscoring the critical role of dedicated modules that leverage historical interaction data for personalization.
📝 Abstract
LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives. Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories. We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from action logs, pairing underspecified intents with rich histories and user-viewed candidate items. Evaluating state-of-the-art LLMs with multi-step agent workflows, we find that models handle explicit queries well but struggle with early-stage queries requiring intent and preference discovery. Rubric analysis attributes this gap mainly to ineffective history use. A simple history-aware query-refinement pipeline, VQRA, yields consistent gains, highlighting the need for dedicated history-utilization modules in personalized agents.
Innovation
Methods, ideas, or system contributions that make the work stand out.
personalization
large language model agents
benchmark
query refinement
interaction history
🔎 Similar Papers
No similar papers found.
