Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems

2026-05-05

AI Summary

Existing user simulators struggle to reproduce realistic human behaviors—such as hesitation and decision delays—under choice overload due to the absence of explicit modeling of human decision-making processes, leading to distorted evaluations of recommender systems. To address this limitation, this work proposes Hesitator, a novel framework that, for the first time, integrates psychological theories of choice overload into user simulator design. Hesitator employs a modular architecture that decouples utility-based preference judgment from commitment decisions influenced by cognitive load. The framework is compatible with various large language models and conversational recommender systems. Experimental results demonstrate that Hesitator consistently reduces unrealistically high acceptance rates across multiple domains and selling contexts, effectively replicates established behavioral patterns, and thereby enhances the ecological validity of recommender system evaluations.

Abstract

Conversational recommender systems (CRS) increasingly rely on user simulators for automated evaluation of sales agents. A key requirement for such simulators is the ability to model human decision-making. However, most existing simulation frameworks do not explicitly model the internal decision process, and LLM-based simulators often exhibit unrealistically strong information-processing capabilities, rarely exhibit the hesitation or decision deferral commonly observed in real consumer behavior, resulting in overly high acceptance probabilities. To address this limitation, we propose Hesitator, a theory-grounded user simulation framework that explicitly models human decision-making under choice overload. The framework introduces a modular Decision Module that separates utility-based item selection from overload-aware commitment decisions. Experiments across multiple user simulation frameworks, domains, sales modes, and LLM backbones show that integrating our module consistently mitigates unrealistic behaviors under increasing overload conditions. Furthermore, Hesitator reproduces established behavioral patterns from psychological economics, demonstrating its ability to model human decision behavior.

Problem

• conversational recommender systems
• user simulation
• decision-making
• choice overload
• behavioral modeling

Innovation

• user simulation
• conversational recommender systems
• decision modeling
• choice overload
• behavioral realism