COMPASS: Computational Mapping of Patient-Therapist Alliance Strategies with Language Modeling

2024-02-22

arXiv.org

AI Summary

Traditional assessment of the working alliance in psychotherapy relies heavily on post-session questionnaires, limiting real-time, fine-grained, and clinically interpretable monitoring. Method: We propose the first framework aligning large language models (LLMs) with distributed representations of established alliance scales, integrating prompt engineering, BERTopic and dynamic topic modeling, and semantic similarity matching. Evaluated on 950+ authentic therapy sessions across diagnostic categories (anxiety, depression, schizophrenia, suicidal ideation), it enables session-level, longitudinal tracking of alliance trajectories. Contribution/Results: The method supports dynamic evolutionary analysis and generates clinically interpretable feedback, revealing disorder-specific dialogue patterns for the first time. It delivers a real-time, objective, and clinically actionable tool for assessing therapeutic relationship quality—substantially enhancing the objectivity, timeliness, and practical utility of alliance monitoring in clinical practice.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The therapeutic working alliance is a critical factor in predicting the success of psychotherapy treatment. Traditionally, working alliance assessment relies on questionnaires completed by both therapists and patients. In this paper, we present COMPASS, a novel framework to directly infer the therapeutic working alliance from the natural language used in psychotherapy sessions. Our approach utilizes advanced large language models (LLMs) to analyze transcripts of psychotherapy sessions and compare them with distributed representations of statements in the working alliance inventory. Analyzing a dataset of over 950 sessions covering diverse psychiatric conditions including anxiety, depression, schizophrenia, and suicidal tendencies, we demonstrate the effectiveness of our method in providing fine-grained mapping of patient-therapist alignment trajectories and offering interpretability for clinical psychiatry and in identifying emerging patterns related to the condition being treated. By employing various deep learning-based topic modeling techniques in combination with prompting generative language models, we analyze the topical characteristics of different psychiatric conditions and their evolution at a turn-level resolution. This combined framework enhances the understanding of therapeutic interactions, enabling timely feedback for therapists regarding the quality of therapeutic relationships and providing interpretable insights to improve the effectiveness of psychotherapy.

Problem

• Infer therapeutic alliance from psychotherapy session language
• Map session dialogues to psychometric instrument representations
• Analyze topic evolution in psychiatric condition conversations

Innovation

• Uses LLMs to analyze therapy session transcripts
• Maps dialogues to psychometric instrument representations
• Combines topic modeling with generative language models