Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming

2026-02-23

AI Summary

Current safety evaluations struggle to identify complex, long-term clinical risks posed by large language models in mental health support—such as reinforcing delusions or failing to detect suicidal ideation. This work proposes the first automated red-teaming framework that integrates dynamic cognitive-affective patient simulation with a clinical risk ontology to enable longitudinal, fine-grained safety auditing of AI-driven psychotherapy through multi-turn dialogue simulations. The framework is complemented by an interactive visualization tool to facilitate multi-stakeholder assessment. In 369 simulated interactions, it successfully uncovered critical vulnerabilities, including “AI-induced psychosis” and failures in suicide risk management. These findings were validated by nine interdisciplinary experts, demonstrating the framework’s effectiveness in exposing black-box risks inherent in AI-based psychological interventions.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Large Language Models (LLMs) are increasingly utilized for mental health support; however, current safety benchmarks often fail to detect the complex, longitudinal risks inherent in therapeutic dialogue. We introduce an evaluation framework that pairs AI psychotherapists with simulated patient agents equipped with dynamic cognitive-affective models and assesses therapy session simulations against a comprehensive quality of care and risk ontology. We apply this framework to a high-impact test case, Alcohol Use Disorder, evaluating six AI agents (including ChatGPT, Gemini, and Character.AI) against a clinically-validated cohort of 15 patient personas representing diverse clinical phenotypes. Our large-scale simulation (N=369 sessions) reveals critical safety gaps in the use of AI for mental health support. We identify specific iatrogenic risks, including the validation of patient delusions ("AI Psychosis") and failure to de-escalate suicide risk. Finally, we validate an interactive data visualization dashboard with diverse stakeholders, including AI engineers and red teamers, mental health professionals, and policy experts (N=9), demonstrating that this framework effectively enables stakeholders to audit the "black box" of AI psychotherapy. These findings underscore the critical safety risks of AI-provided mental health support and the necessity of simulation-based clinical red teaming before deployment.

Problem

• Large Language Models
• Mental Health Support
• AI Safety
• Clinical Red Teaming
• Iatrogenic Risk

Innovation

• AI red teaming
• simulated patient agents
• dynamic cognitive-affective models
• risk ontology
• interactive visualization dashboard