Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

2026-07-11

AI Summary

Current single-turn evaluation benchmarks inadequately assess large language models’ capabilities in maintaining persona consistency, tracking user intent, managing emotional dynamics, and achieving conversational goals across multi-turn dialogues. To address this gap, this work proposes EYT-Bench, a benchmark featuring a tripartite decoupled architecture comprising a user simulator grounded in real-world persona corpora (Nemotron-Personas-USA and PersonaMem-v2), a goal-oriented model that separates intent awareness from response generation, and an ensemble裁判 mechanism leveraging multiple LLMs. The framework introduces trajectory-level metrics—such as intent drift and Final Intent Completion Rate (FICR)—alongside embedded intent tracking. Evaluation across 17 models and 200 dialogues reveals minimal subjective score variation (≤0.3) but up to a ninefold disparity in objective intent-tracking performance; reasoning mechanisms substantially improve intent accuracy over long contexts; persona representation format significantly influences goal achievement; and a warm-start effect robustly persists in 16 out of 17 models.

Abstract

Evaluating large language models (LLMs) as multi-turn conversational partners requires probing capabilities that single-turn benchmarks miss: persona consistency, evolving intent tracking, emotional dynamics, and goal completion. We introduce EYT-Bench, a human-centered benchmark built around a three-party decoupled design: a persona-grounded user simulator, a target model that separates intent perception from response generation, and an independent third-party LLM judge with optional multi-judge ensembling. Personas are sampled from public human-curated corpora, Nemotron-Personas-USA and PersonaMem-v2, rather than synthesized, reducing LLM-induced persona bias. EYT-Bench also introduces two trajectory-level metrics: embedding-based intent drift and final-intent completion rate (FICR), inspired by tau-bench. In a 17-target x 200-dialogue evaluation, EYT-Bench reveals four findings: (i) state-of-the-art closed- and open-source models are statistically close on subjective dimensions (empathy / persona / anthropomorphism vary within <= 0.3), but differ by up to 9x on objective intent tracking; (ii) reasoning ("thinking on") sharply improves objective tracking on long-context personas (+0.47-0.50 latent-intent accuracy on Gemma-4) while leaving subjective scores nearly unchanged; (iii) persona format dominates trajectory spread, with FICR saturating above 0.95 on Nemotron-USA but spreading from 0.53 to 0.88 on PersonaMem-v2; and (iv) the warm-up effect is robust on 16/17 models (one outlier, GPT-5.5, reverses the effect), with stable rankings across alpha in [0.05, 0.15]. A cross-judge ablation using deepseek-v4-pro confirms that target rankings and final-intent satisfaction are preserved across judges.

Problem

• multi-turn dialogue
• persona consistency
• intent tracking
• emotional dynamics
• goal completion

Innovation

• decoupled user simulation
• intent tracking
• trajectory-level metrics
• persona consistency
• multi-turn dialogue benchmark