Socratic Reasoning Improves Positive Text Rewriting

2024-03-05

arXiv.org

AI Summary

Existing approaches to positively reframing negative mental health texts lack sufficient multi-step rationalization modeling. Method: We propose SocraticReframe—a novel framework that introduces the Socratic questioning chain (questioning → reflection → reconstruction) into psychotherapeutic text rewriting, enabling interpretable and clinically credible cognitive restructuring. Leveraging open-source large language models (LLMs), we generate high-quality rationale data, which is then rigorously annotated and validated by psychology experts under IRB approval, yielding the first synthetic dataset dedicated to cognitive restructuring. Results: Experiments demonstrate significant improvements across both automated metrics and therapy-oriented human evaluation for multiple open-source LLMs; expert acceptance reaches 89.2%. Our work bridges critical gaps in both multi-step reasoning–driven psychological text rewriting modeling and associated training data infrastructure.

Technology Category

• Natural Language Processing: Safety and Robustness
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Machine Learning: Feature Construction/Reformulation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reframing a negative into a positive thought is at the crux of several cognitive approaches to mental health and psychotherapy that could be made more accessible by large language model-based solutions. Such reframing is typically non-trivial and requires multiple rationalization steps to uncover the underlying issue of a negative thought and transform it to be more positive. However, this rationalization process is currently neglected by both datasets and models which reframe thoughts in one step. In this work, we address this gap by augmenting open-source datasets for positive text rewriting with synthetically-generated Socratic rationales using a novel framework called extsc{SocraticReframe}. SocraticReframe uses a sequence of question-answer pairs to rationalize the thought rewriting process. We show that such Socratic rationales significantly improve positive text rewriting for different open-source LLMs according to both automatic and human evaluations guided by criteria from psychotherapy research. We validate our framework and the synthetic rationalizations with expert judgements from domain experts and psychology students in an IRB-approved annotation study. Our findings highlight the potential of utilizing the synergy between LLM reasoning and established psychotherapy techniques to build assistive solutions for reframing negative thoughts.

Problem

• Improving positive text rewriting using Socratic reasoning
• Addressing the gap in rationalization steps for thought reframing
• Enhancing LLM-based solutions for mental health psychotherapy

Innovation

• Synthetic Socratic rationales enhance text rewriting
• SocraticReframe uses question-answer pairs for rationalization
• LLM reasoning combined with psychotherapy techniques