Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations

2026-05-04

AI Summary

This work addresses the limitations of existing user simulation methods, which often fail to capture the complexity and communicative friction inherent in real human–agent interactions, leading to overly optimistic evaluation results. The authors propose Realsim, a novel evaluation framework that systematically compares real and simulated multi-turn dialogues across eight dimensions—including communicative functions, user states, and surface linguistic forms—from a distributional perspective. To support this analysis, they introduce a dataset of 1,000 real conversations spanning 16 domains. Through multidimensional distributional comparisons, newly designed evaluation metrics, and empirical analysis, the study reveals that current simulators generally struggle to reproduce communicative friction and exhibit significant performance disparities across domains, thereby underscoring the need for domain-adapted user simulators.

Abstract

There is growing interest in exploring user simulation as an alternative to gathering and scoring real user-chatbot interactions for AI chatbot evaluation. For this purpose, it is important to ensure the realism of the simulation, i.e., the extent to which simulated dialogues reflect real dialogues users have with chatbots. Most existing methods evaluating simulation realism produce coarse quality signal and remain solely at the level of individual dialogues. To support more rigorous evaluation in this area, we propose realsim, an evaluation framework that enables practitioners to take a distributional view of real vs. simulated dialogues along 8 dimensions, covering attributes related to the communicative functions of the interaction, user states, and the surface form of user messages. We then instantiate the framework with a curated dataset of 1K multi-turn task-focused real user-chatbot dialogues that cover 16 domains of chatbot applications. Overall, we find that simulated users tend to struggle at capturing communication frictions that real users introduce to interactions, which could make evaluations based on such simulations overly optimistic. We also observe variability in performance across different domains, which may indicate a need for domain-specific user simulators.

Problem

• user simulation
• dialogue realism
• multi-turn conversations
• chatbot evaluation
• distributional differences

Innovation

• user simulation
• evaluation framework
• dialogue realism
• multi-turn conversations
• distributional comparison