Mitigating Semantic Drift: Evaluating LLMs' Efficacy in Psychotherapy through MI Dialogue Summarization

2025-11-27

AI Summary

This study addresses critical limitations of large language models (LLMs) in low-resource, sensitive domains—specifically, psychotherapy summarization using Motivational Interviewing (MI)—including insufficient sensitivity, factual inaccuracies, inconsistent empathic expression, bias, and hallucination. To this end, we propose a fine-grained evaluation framework grounded in the Motivational Interviewing Treatment Integrity (MITI) coding system. A high-quality MI dialogue summarization dataset was constructed via two-stage human annotation; subsequently, a multi-class classification task—integrated with both zero-shot and few-shot prompting strategies—was designed to systematically assess LLMs’ comprehension of core therapeutic constructs: empathy, collaboration, and autonomy support. Our empirical analysis first demonstrates that progressive prompting significantly mitigates semantic drift and enhances contextual fidelity. The work not only empirically delineates the capabilities and boundaries of LLMs in clinical summarization but also establishes a reproducible evaluation benchmark and provides practical, domain-informed prompt engineering guidelines for sensitive applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Summarization
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advancements in large language models (LLMs) have shown their potential across both general and domain-specific tasks. However, there is a growing concern regarding their lack of sensitivity, factual incorrectness in responses, inconsistent expressions of empathy, bias, hallucinations, and overall inability to capture the depth and complexity of human understanding, especially in low-resource and sensitive domains such as psychology. To address these challenges, our study employs a mixed-methods approach to evaluate the efficacy of LLMs in psychotherapy. We use LLMs to generate precise summaries of motivational interviewing (MI) dialogues and design a two-stage annotation scheme based on key components of the Motivational Interviewing Treatment Integrity (MITI) framework, namely evocation, collaboration, autonomy, direction, empathy, and a non-judgmental attitude. Using expert-annotated MI dialogues as ground truth, we formulate multi-class classification tasks to assess model performance under progressive prompting techniques, incorporating one-shot and few-shot prompting. Our results offer insights into LLMs' capacity for understanding complex psychological constructs and highlight best practices to mitigate ``semantic drift" in therapeutic settings. Our work contributes not only to the MI community by providing a high-quality annotated dataset to address data scarcity in low-resource domains but also critical insights for using LLMs for precise contextual interpretation in complex behavioral therapy.

Problem

• Evaluating LLMs' ability to summarize psychotherapy dialogues accurately
• Assessing LLMs' understanding of complex psychological constructs like empathy
• Mitigating semantic drift in LLMs for sensitive therapeutic applications

Innovation

• Using LLMs to summarize motivational interviewing dialogues
• Designing two-stage annotation scheme based on MITI framework
• Employing progressive prompting techniques for multi-class classification