Learning User Simulators with Turing Rewards

2026-06-17

AI Summary

This work proposes Turing-RL, a novel approach to enhancing the realism of user simulators for agent training and system evaluation by incorporating the core idea of the Turing test into simulator learning. The method employs a large language model–based generator and discriminator trained via reinforcement learning to optimize indistinguishability—ensuring simulated responses are difficult to differentiate from genuine user behavior—rather than merely aligning with reference answers on surface-level metrics. Evaluated in both conversational chat and Reddit forum settings, Turing-RL consistently outperforms existing baselines across automatic metrics and human assessments, demonstrating its effectiveness and innovation in generating authentic user-like interactions.

Abstract

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose {Turing-RL}: a Turing-Test-based reinforcement learning approach for training user simulator models. {Turing-RL} uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that {Turing-RL} consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.

Problem

• user simulation
• Turing test
• indistinguishability
• interactive systems
• large language models

Innovation

• Turing-RL
• user simulation
• reinforcement learning
• Turing reward
• large language models