UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs

2026-08-10

AI Summary

Existing benchmarks for tool-augmented large language models primarily focus on user profile memorization or stylistic imitation, overlooking the evaluation of personalized decision-making under incomplete information. This work introduces the first evaluation paradigm that conceals user profiles, leveraging privacy-preserving real-world interaction data to assess a model’s ability to infer user preferences, recognize when clarification is needed, and generate tool-call sequences aligned with user intent despite missing information. The framework incorporates structured persona modeling, simulated multi-tool APIs, and multi-turn, long-horizon task designs. Experimental results reveal that state-of-the-art models still struggle with personalized delegation tasks, with key bottlenecks lying in multi-tool coordination, constrained reasoning, and consistency in long-term behavior.

Abstract

Tool-use LLMs are increasingly asked to act on users' behalf, but existing benchmarks usually focus on profile recall, style imitation, generic tool use, or response-level personalization. We introduce UserToolBench , a benchmark for personalized decision making in tool-use LLMs. UserToolBench tests whether a model can infer latent user preferences from interaction history, recognize when clarification is needed, and produce user-aligned tool-call trajectories under incomplete information. The benchmark is built from privacy-sanitized real interaction traces and combines structured persona profiles, public API-style tool ecosystems, and long-horizon multi-turn trajectories. It includes 10 user profiles, 36 tool sets, 1,065 turns, 170 unique tools, and evaluation-focused task types covering lack-of-information, single-tool, and multi-tool settings. Experiments with strong tool-use LLMs show that current models still have difficulty with personalized delegation. Multi-tool coordination, missing-constraint inference, and long-horizon behavioral consistency remain major bottlenecks. These results suggest that personalization evaluation should move beyond asking whether outputs sound user-specific and instead ask whether LLMs make correct decisions for the users they represent.

Problem

• personalized decision making
• tool-use LLMs
• user preference inference
• incomplete information
• behavioral consistency

Innovation

• personalized decision making
• tool-use LLMs
• user preference inference
• privacy-sanitized benchmark
• multi-tool coordination