When Symptoms Are Not Enough: Evidence-Weighting Patterns in Large Language Model Psychiatric Screening

2026-05-21

AI Summary

Current large language models (LLMs) lack a clear mechanism for integrating multidimensional evidence—such as symptomatology, functional impairment, and protective contextual factors—in mental health screening, leading to insufficient reliability across diagnostic categories and demographic subgroups. This study constructs a benchmark dataset of 555 semi-structured interviews aligned with the SCID gold standard and evaluates multiple state-of-the-art LLMs in zero-shot, task-specific prompting for screening anxiety disorders, depression, and PTSD. It reveals, for the first time, that LLMs systematically over-rely on indicators of functional intactness or protective context while overlooking explicit symptom evidence, thereby increasing the risk of false negatives. Results show that GPT-4.1 Mini and GPT-5 Mini achieve the most consistent performance (accuracy: 0.49–0.86), with better depression detection in males than females, and that functional impairment strongly drives positive classifications. The findings underscore the necessity of rigorous validation before clinical deployment of LLMs.

Abstract

As demand for mental health care outpaces clinician-delivered assessment, scalable screening tools are increasingly needed. Large language models (LLMs) may identify psychiatric risk from patient narratives, but their reliability across diagnoses, demographic subgroups, and evidence-use patterns remains uncertain. We introduce a SCID-anchored benchmark of 555 semi-structured experiential interviews paired with diagnostic reference labels for anxiety disorder, major depressive disorder, post-traumatic stress disorder, and any current mental health disorder. Using zero-shot task-specific prompting, we evaluated five state-of-the-art LLMs and examined whether false-negative errors reflected missed psychiatric evidence or differential weighting of symptom, functional-impairment, and protective-context cues. Performance varied across tasks and models, with accuracy ranging from 0.49 to 0.86 and Matthews correlation coefficients from 0.16 to 0.38. GPT-4.1 Mini and GPT-5 Mini showed the most consistent disorder-specific accuracy. Subgroup analyses found higher depression-classification accuracy among male than female participants, no consistent age-related pattern, and modest non-uniform variation across race strata. Evidence-integration analyses showed that false-negative anxiety and PTSD classifications often contained explicit symptom evidence but were accompanied by preserved functioning, coping ability, or social support. Functional-impairment evidence shifted model outputs toward positive classifications, whereas protective-context evidence shifted outputs away. These findings suggest that LLMs may support scalable psychiatric screening, but their tendency to discount symptom evidence in the presence of preserved functioning or protective context requires careful validation before clinical deployment.

Problem

• large language models
• psychiatric screening
• evidence weighting
• diagnostic reliability
• false-negative errors

Innovation

• evidence-weighting
• large language models
• psychiatric screening
• functional impairment
• false-negative analysis