Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning

2026-04-15

AI Summary

This work addresses the lack of effective evaluation methods for multimodal consistency between speech and character attributes in role-playing dialogue systems, particularly the absence of quantitative measures for paralinguistic cues. To this end, the authors propose RoleJudge, a multidimensional evaluation framework that leverages audio large language models to assess alignment between spoken utterances and target roles. They also introduce RoleChat, the first speech-based role-playing dataset annotated with chain-of-thought reasoning. Furthermore, a Standard Alignment mechanism is incorporated into reinforcement learning training to mitigate reward misalignment. Experimental results demonstrate that the proposed approach significantly outperforms baseline methods in both objective accuracy and subjective evaluations, confirming the effectiveness and robustness of the framework.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Conversational AI/Dialog Systems
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The rapid evolution of multimodal large models has revolutionized the simulation of diverse characters in speech dialogue systems, enabling a novel interactive paradigm. Character attributes are manifested not only in textual responses but also through vocal features, as speech conveys rich paralinguistic information that is challenging to quantify. This poses significant difficulties in evaluating the character alignment of role-playing agents. To address these challenges, we present RoleJudge, an evaluation framework that leverages audio large language models to systematically assess the alignment between speech and character across multiple modalities and dimensions. Furthermore, we introduce RoleChat, the first voice role-playing evaluation dataset enriched with chain-of-thought reasoning annotations, comprising a diverse set of authentic and LLM-generated speech samples. Utilizing this dataset, we implement a multi-stage training paradigm and incorporate Standard Alignment in reinforcement learning to mitigate reward misalignment during optimization. Experimental results in terms of accuracy and subjective assessment demonstrate that RoleJudge outperforms various baseline models, validating the effectiveness of our multidimensional evaluation framework.

Problem

• character alignment
• role-playing evaluation
• audio large language models
• paralinguistic information
• multimodal evaluation

Innovation

• audio large language models
• role-playing evaluation
• character alignment
• reinforcement learning
• multimodal evaluation