RedditESS: A Mental Health Social Support Interaction Dataset -- Understanding Effective Social Support to Refine AI-Driven Support Tools

2025-03-27

AI Summary

Prior research overemphasizes empathic expression while neglecting critical dimensions of mental health support—namely informational guidance, group identification, and actionable coping strategies. Method: We introduce RedditESS, the first multidimensionally validated dataset derived from authentic Reddit depression-help-seeking interactions, comprising supportive comments and corresponding help-seekers’ feedback. Grounded in sociological theory and employing a multi-expert collaborative annotation protocol, we systematically define and label informational, identificational, and strategic support dimensions. We propose a novel validity evaluation paradigm using help-seekers’ subsequent responses as the gold standard. Using RLHF/PPO-based alignment fine-tuning and qualitative content analysis, we enhance LLMs’ contextual sensitivity and practical utility. Results: Human evaluation shows a 37% increase in effective support rate. This work establishes an interpretable, empirically grounded, multidimensional benchmark and methodological framework for AI-enabled psychological intervention.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Effective mental health support is crucial for alleviating psychological distress. While large language model (LLM)-based assistants have shown promise in mental health interventions, existing research often defines"effective"support primarily in terms of empathetic acknowledgments, overlooking other essential dimensions such as informational guidance, community validation, and tangible coping strategies. To address this limitation and better understand what constitutes effective support, we introduce RedditESS, a novel real-world dataset derived from Reddit posts, including supportive comments and original posters' follow-up responses. Grounded in established social science theories, we develop an ensemble labeling mechanism to annotate supportive comments as effective or not and perform qualitative assessments to ensure the reliability of the annotations. Additionally, we demonstrate the practical utility of RedditESS by using it to guide LLM alignment toward generating more context-sensitive and genuinely helpful supportive responses. By broadening the understanding of effective support, our study paves the way for advanced AI-driven mental health interventions.

Problem

• Defining effective mental health support beyond empathy
• Creating a dataset to improve AI support tools
• Enhancing LLM responses for mental health interventions

Innovation

• RedditESS dataset for real-world mental health support
• Ensemble labeling to annotate effective support comments
• LLM alignment for context-sensitive supportive responses