Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants

2026-05-10

AI Summary

This work addresses the challenge of effectively evaluating user simulators employed to train collaborative large language model (LLM) assistants. It proposes using downstream task utility derived from real human interactions as the primary evaluation criterion and systematically compares role-playing prompts against simulators fine-tuned on authentic dialogues from WildChat. LLM assistants trained via reinforcement learning with these simulators are evaluated on the WildBench benchmark and through large-scale user studies. Results demonstrate that assistants trained with fine-tuned simulators achieve a 58% win rate in real-user evaluations, significantly outperforming those trained with role-playing methods (51%) and exhibiting stronger generalization capabilities. These findings validate the efficacy of data-driven user simulators in enhancing the collaborative performance of LLMs.

Abstract

User simulators are increasingly leveraged to build interactive AI assistants, yet how to measure the quality of these simulators remains an open question. In this work, we show how simulator quality can be quantified in terms of its downstream utility: how an LLM assistant trained with this user simulator performs in the wild when interacting with real humans. In a controlled experiment where only the user simulator varies, we train LLM assistants via reinforcement learning against a spectrum of simulators, from an LLM prompted to role-play a user to one fine-tuned on human utterances from WildChat. As evaluation, we measure pairwise win rates in a user study with 283 participants and on WildBench, a benchmark derived from real human--AI conversations. Training against the role-playing LLM yields an assistant statistically indistinguishable from the initial assistant in our user study (51% win rate), whereas training against the fine-tuned simulator yields significant gains (58% over the initial and 57% over the one trained against role-playing). Closer inspection reveals three further patterns: methods for making role-playing LLMs more realistic (e.g., persona conditioning) improve trained assistants but do not close the gap to the fine-tuned simulator; scaling the simulator's model size benefits the fine-tuned simulator but yields no gain for role-playing ones; and assistants trained against role-playing simulators fail to generalize when paired with other simulators at test time, while the one trained against fine-tuned simulator does. Together, these results argue for grounding user simulators in real human behavior and measuring their quality by their downstream effect on real users.

Problem

• user simulators
• LLM assistants
• simulator quality
• downstream utility
• human-AI interaction

Innovation

• user simulators
• downstream utility
• LLM assistants
• reinforcement learning
• human-AI interaction