Evaluating Role-Consistency in LLMs for Counselor Training

2026-01-13

AI Summary

This study addresses the challenge of insufficient role consistency and dialogue coherence exhibited by large language models (LLMs) when simulating client roles in online psychological counseling training. To systematically evaluate model performance, the authors introduce a novel role-consistency evaluation dataset constructed using adversarial examples specifically tailored to counseling scenarios. Leveraging metrics that jointly assess role fidelity and conversational coherence, the work provides a comprehensive benchmark of several open-source LLMs, including Vicuna. The findings not only quantify disparities in role-maintenance capabilities across models but also offer empirical guidance for selecting appropriate LLMs in virtual counseling training systems. Furthermore, this research highlights the innovative value of adversarial evaluation methodologies in role-playing tasks within specialized domains such as mental health counseling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rise of online counseling services has highlighted the need for effective training methods for future counselors. This paper extends research on VirCo, a Virtual Client for Online Counseling, designed to complement traditional role-playing methods in academic training by simulating realistic client interactions. Building on previous work, we introduce a new dataset incorporating adversarial attacks to test the ability of large language models (LLMs) to maintain their assigned roles (role-consistency). The study focuses on evaluating the role consistency and coherence of the Vicuna model's responses, comparing these findings with earlier research. Additionally, we assess and compare various open-source LLMs for their performance in sustaining role consistency during virtual client interactions. Our contributions include creating an adversarial dataset, evaluating conversation coherence and persona consistency, and providing a comparative analysis of different LLMs.

Problem

• role-consistency
• large language models
• virtual counseling
• adversarial attacks
• persona consistency

Innovation

• role-consistency
• adversarial attacks
• large language models
• virtual counseling
• persona consistency