Preference Learning Unlocks LLMs' Psycho-Counseling Skills

2025-02-27

AI Summary

Addressing two critical challenges in AI-powered psychological counseling—scarcity of high-quality real-world data and absence of standardized response evaluation criteria—this work proposes a systematic solution. First, we introduce PsychoCounsel-Preference, the first therapist-aligned, high-fidelity, privacy-compliant preference dataset for psychotherapeutic dialogues (36K preference pairs). Second, we establish the first psychology-grounded response evaluation framework, integrating Direct Preference Optimization (DPO) and reward modeling to enable unsupervised, real-conversation-based alignment of LLMs with clinical principles. Third, building upon Llama3-8B, we conduct supervised fine-tuning followed by RLHF optimization, releasing PsychoCounsel-Llama3-8B, which achieves an 87% win rate over GPT-4o in adversarial benchmarking. We publicly release the dataset, model, and reward model to advance trustworthy, clinically informed AI for mental health support.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Applying large language models (LLMs) to assist in psycho-counseling is an emerging and meaningful approach, driven by the significant gap between patient needs and the availability of mental health support. However, current LLMs struggle to consistently provide effective responses to client speeches, largely due to the lack of supervision from high-quality real psycho-counseling data, whose content is typically inaccessible due to client privacy concerns. Furthermore, the quality of therapists' responses in available sessions can vary significantly based on their professional training and experience. Assessing the quality of therapists' responses remains an open challenge. In this work, we address these challenges by first proposing a set of professional and comprehensive principles to evaluate therapists' responses to client speeches. Using these principles, we create a preference dataset, PsychoCounsel-Preference, which contains 36k high-quality preference comparison pairs. This dataset aligns with the preferences of professional psychotherapists, providing a robust foundation for evaluating and improving LLMs in psycho-counseling. Experiments on reward modeling and preference learning demonstrate that PsychoCounsel-Preference is an excellent resource for LLMs to acquire essential skills for responding to clients in a counseling session. Our best-aligned model, PsychoCounsel-Llama3-8B, achieves an impressive win rate of 87% against GPT-4o. We release PsychoCounsel-Preference, PsychoCounsel-Llama3-8B and the reward model PsychoCounsel Llama3-8B-Reward to facilitate the research of psycho-counseling with LLMs at: https://hf.co/Psychotherapy-LLM.

Problem

• Enhance LLMs' psycho-counseling response quality
• Create preference dataset for therapist evaluation
• Improve LLMs' skills using PsychoCounsel-Preference dataset

Innovation

• Preference dataset for psycho-counseling
• Reward modeling with professional principles
• Aligned LLMs outperform GPT-4o