Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors

2026-05-08

AI Summary

Existing large language model–based user simulators struggle to capture the diversity and heterogeneity of real user behaviors. This work proposes a systematic evaluation framework grounded in behavioral representation learning, clustering quantification, and distributional divergence measurement. It presents the first quantitative analysis of behavioral distribution biases across 24 user simulators, employing TF-IDF–enhanced interpretable clustering to uncover distinct patterns in captured, missing, and hallucinated behaviors. Building on these insights, the study further introduces a behavior-complementarity–based fusion strategy that effectively narrows the gap between simulated and real user behavior distributions, substantially improving simulation fidelity.

Abstract

As user simulators are increasingly used for interactive training and evaluation of AI assistants, it is essential that they represent the diverse behaviors of real users. While existing works train user simulators to generate human-like responses, whether they capture the broad and heterogeneous distribution of real user behaviors remains an open question. In this work, we introduce a method to measure the distributional gap between real and simulated user behaviors, validated through a human study and ablations. Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation, quantizes them into discrete distributions via clustering, then computes divergence metrics. We provide the first systematic evaluation of 24 LLM-based user simulators on coding and writing tasks, and reveal a large distributional gap from real users that varies across model families, scales, and behavioral facets. Pairwise comparisons show that most simulators behave similarly, while a few stand apart. Combining behaviorally complementary simulators brings the resulting distribution closer to real users compared to either simulator on its own. Finally, a TF-IDF analysis of the clusters surfaces interpretable patterns of behaviors that simulators capture, miss, and hallucinate.

Problem

• user simulators
• distributional gap
• real user behaviors
• simulated behaviors
• behavioral heterogeneity

Innovation

• distributional gap
• user simulators
• behavior representation
• clustering-based quantization
• LLM evaluation