Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice

2025-01-02

AI Summary

This study investigates the feasibility of leveraging large language models (LLMs) for unsupervised mental health screening, specifically depression detection and automated scoring of standardized clinical instruments (e.g., BDI-II). We propose the first framework that tightly integrates adaptive retrieval-augmented generation (RAG) with classical psychometric principles: given social media text, it performs semantic retrieval to align scale items with authentic user expressions, enabling zero-shot, item-level, traceable scale completion and clinical pathway inference. The method requires no fine-tuning or labeled data, balancing predictive accuracy with clinical interpretability. Evaluated on a Reddit benchmark dataset, our approach achieves state-of-the-art accuracy in BDI-II scoring and supports DSM-5–compliant diagnostic reasoning tracing. This work establishes a novel paradigm for low-cost, high-fidelity digital mental health triage.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In psychological practice, standardized questionnaires serve as essential tools for assessing mental constructs (e.g., attitudes, traits, and emotions) through structured questions (aka items). With the increasing prevalence of social media platforms where users share personal experiences and emotions, researchers are exploring computational methods to leverage this data for rapid mental health screening. In this study, we propose a novel adaptive Retrieval-Augmented Generation (RAG) approach that completes psychological questionnaires by analyzing social media posts. Our method retrieves the most relevant user posts for each question in a psychological survey and uses Large Language Models (LLMs) to predict questionnaire scores in a zero-shot setting. Our findings are twofold. First we demonstrate that this approach can effectively predict users' responses to psychological questionnaires, such as the Beck Depression Inventory II (BDI-II), achieving performance comparable to or surpassing state-of-the-art models on Reddit-based benchmark datasets without relying on training data. Second, we show how this methodology can be generalized as a scalable screening tool, as the final assessment is systematically derived by completing standardized questionnaires and tracking how individual item responses contribute to the diagnosis, aligning with established psychometric practices.

Problem

• Large Language Models
• Mental Health Screening
• Depression Detection

Innovation

• Adaptive RAG
• Psychological Assessment
• Social Media Analysis