MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters

2026-07-09

AI Summary

Existing benchmarks inadequately assess the capabilities of large language models (LLMs) in real-world psychiatric clinical practice. This work introduces the first end-to-end virtual clinical evaluation environment encompassing all major ICD-11 diagnostic categories, generating standardized patients from 1,193 de-identified electronic health records and adhering to the S.O.A.P. (Subjective, Objective, Assessment, Plan) framework. We propose MentalEval, a scalable expert-aligned evaluation framework that integrates five domain-specific evaluators—each refined through rule-based fine-tuning and expert-guided Direct Preference Optimization—to assess empathetic communication, clinical interviewing proficiency, clinical note quality, diagnostic rigor, and treatment appropriateness. Experiments demonstrate that MentalEval achieves high agreement with ratings from 22 clinicians (mean quadratic weighted kappa = 0.944), while even the strongest current LLMs lag behind human clinicians by 37.28 percentage points in objective psychiatric competence.

Abstract

Large language models (LLMs) have shown strong performance on isolated psychiatric tasks, including dialogue, diagnosis, and treatment planning, yet existing benchmarks rarely simulate complete psychiatric clinical encounters. We introduce $\textbf{MentalHospital}$, a virtual evaluation environment for LLM-based psychiatric clinical encounters. MentalHospital instantiates the Subjective Interviewing, Objective Examination, Diagnostic Assessment, and Treatment Planning (S.O.A.P.) workflow, using skill-augmented standardized patients constructed from 1,193 de-identified psychiatric electronic health record (EHR) cases spanning all major ICD-11 categories and 76 disorders. Each encounter is assessed through a dual-track protocol that combines objective comparison against EHR-derived references with subjective assessment of clinical process quality. To scale specialist judgment, we develop $\textbf{MentalEval}$, five domain-specific evaluators covering communication empathy, interviewing professionalism, clinical-note quality, diagnostic rigor, and treatment appropriateness, trained with rubric-grounded SFT and expert-guided DPO. Survey responses from 22 clinicians support MentalHospital's clinical fidelity (3.88/5), while MentalEval achieves strong expert alignment with an average QWK of 0.944. Benchmarking shows that even the strongest LLM trails clinicians by 37.28 percentage points in objective psychiatric competence, with mental status assessment as a key bottleneck.

Problem

• psychiatric clinical encounters
• large language models
• clinical evaluation
• mental health assessment
• benchmarking

Innovation

• MentalHospital
• psychiatric clinical evaluation
• S.O.A.P. workflow
• MentalEval
• large language models