Multi-dimensional Assessment and Explainable Feedback for Counselor Responses to Client Resistance in Text-based Counseling with LLMs

2026-02-25

AI Summary

This study addresses the lack of fine-grained, interpretable evaluation mechanisms for assessing how counselors respond to client resistance—a critical gap that hinders skill development in psychotherapy. To bridge this gap, the authors propose a novel, theory-driven multidimensional evaluation framework that decomposes counselor responses in resistance scenarios into four distinct communication mechanisms. Leveraging expert-annotated data, they perform full-parameter instruction tuning on Llama-3.1-8B-Instruct to jointly model response quality scoring and explanatory rationale generation. The resulting model achieves 77–81% F1 in identifying the quality of different communication mechanisms, significantly outperforming GPT-4o and Claude-3.5-Sonnet. Moreover, its generated explanations receive near-perfect expert ratings (2.8–2.9/3.0) and are empirically validated to effectively enhance the resistance-handling competence of 43 practicing counselors.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Effectively addressing client resistance is a sophisticated clinical skill in psychological counseling, yet practitioners often lack timely and scalable supervisory feedback to refine their approaches. Although current NLP research has examined overall counseling quality and general therapeutic skills, it fails to provide granular evaluations of high-stakes moments where clients exhibit resistance. In this work, we present a comprehensive pipeline for the multi-dimensional evaluation of human counselors' interventions specifically targeting client resistance in text-based therapy. We introduce a theory-driven framework that decomposes counselor responses into four distinct communication mechanisms. Leveraging this framework, we curate and share an expert-annotated dataset of real-world counseling excerpts, pairing counselor-client interactions with professional ratings and explanatory rationales. Using this data, we perform full-parameter instruction tuning on a Llama-3.1-8B-Instruct backbone to model fine-grained evaluative judgments of response quality and generate explanations underlying. Experimental results show that our approach can effectively distinguish the quality of different communication mechanisms (77-81% F1), substantially outperforming GPT-4o and Claude-3.5-Sonnet (45-59% F1). Moreover, the model produces high-quality explanations that closely align with expert references and receive near-ceiling ratings from human experts (2.8-2.9/3.0). A controlled experiment with 43 counselors further confirms that receiving these AI-generated feedback significantly improves counselors' ability to respond effectively to client resistance.

Problem

• client resistance
• counselor feedback
• text-based counseling
• multi-dimensional assessment
• explainable AI

Innovation

• client resistance
• multi-dimensional evaluation
• explainable AI feedback
• instruction tuning
• counseling communication mechanisms