One-shot emergency psychiatric triage across 15 frontier AI chatbots

2026-04-28

AI Summary

This study evaluates the accuracy and reliability of state-of-the-art AI chatbots in triaging psychiatric emergencies during single-turn conversations, addressing the challenge of assessing the urgency of psychological crises in the absence of objective metrics. Using 112 standardized clinical vignettes encompassing nine psychiatric symptom categories and nine risk dimensions, the research tested 15 leading AI models on their ability to assign one of four urgency levels, benchmarked against consensus ratings from 50 psychiatrists as the gold standard. The work introduces the first systematic evaluation framework for AI-based psychiatric triage, revealing that while AI models achieve 94.3% accuracy in identifying cases requiring immediate intervention (Level D), they exhibit significant over-triage for moderate- and low-risk scenarios, resulting in overall accuracy ranging from 42.0% to 71.8%.

Abstract

AI chatbots are increasingly used for health advice, but their performance in psychiatric triage remains undercharacterized. Psychiatric triage is particularly challenging because urgency must often be inferred from thoughts, behavior, and context rather than from objective findings. We evaluated the performance of 15 frontier AI chatbots on psychiatric triage from realistic single-message disclosures using 112 clinical vignettes, each paired with 1 of 4 original benchmark triage labels: A, routine; B, assessment within 1 week; C, assessment within 24 to 48 hours; and D, emergency care now. Vignettes covered 9 psychiatric presentation clusters and 9 focal risk dimensions, organized into 28 presentation-by-risk groups. Each group contributed 4 distinct vignettes, with 1 vignette at each triage level. Each vignette was rendered as a realistic human-authored conversational query, and the AI chatbots were tasked with assigning a triage label from that disclosure. Emergency under-triage occurred in 23 of 410 level D trials (5.6%), and all under-triaged emergencies were reassigned to level C urgency. Across target models, average accuracy ranged from 42.0% to 71.8%. Accuracy was highest for level D vignettes (94.3%) and lowest for level B vignettes (19.7%). Mean signed ordinal error was positive (+0.47 triage levels), indicating net over-triage. Dispersion was highest around the middle triage levels. All results were confirmed relative to clinician consensus labels from 50 medical doctors. When presented with user messages containing sufficient clinical information, frontier AI chatbots thus recognized psychiatric emergencies as requiring urgent medical assessment with near-zero error rates, yet showed marked over-triage for low and intermediate risk presentations.

Problem

• psychiatric triage
• AI chatbots
• emergency under-triage
• one-shot assessment
• mental health urgency

Innovation

• psychiatric triage
• AI chatbots
• one-shot evaluation
• clinical vignettes
• emergency under-triage