🤖 AI Summary
Existing evaluation of role-playing language models suffers from heavy reliance on manual annotation, static single-dimensional metrics, and poor scalability. To address these limitations, this paper proposes the first large-model-driven, dynamic closed-loop evaluation framework. Methodologically, it introduces a tripartite role-based architecture—“Player,” “Inquirer,” and “Judge”—where the model autonomously simulates diverse user dialogue behaviors and enables multi-judge collaborative assessment to automate scoring and alignment of multi-turn interaction quality. Crucially, it establishes an end-to-end evaluation loop without human annotation. Experimental results demonstrate high correlation (Spearman ρ > 0.85) between automated scores and human judgments across key dimensions—including coherence, role consistency, and response appropriateness—while significantly improving scalability, cross-model consistency, and ecosystem compatibility.
📝 Abstract
We introduce a benchmark for evaluating the role-playing capabilities of language models. Our approach leverages language models themselves to emulate users in dynamic, multi-turn conversations and to assess the resulting dialogues. The framework consists of three main components: a player model that assumes a specific character role, an interrogator model that simulates user behavior, and several judge models that evaluate conversation quality. We conducted experiments comparing automated evaluations with human annotations to validate our approach, demonstrating strong correlations across multiple criteria. This work provides a foundation for a robust and dynamic evaluation of the model capabilities in interactive scenarios.