Analyzing LLM Reasoning to Uncover Mental Health Stigma

2026-04-27

AI Summary

This study addresses the pervasive yet often implicit stigmatizing biases embedded in large language models when discussing mental health topics—biases that conventional evaluation methods frequently fail to detect due to their deep-seated logical underpinnings. To tackle this issue, the work introduces chain-of-thought analysis into mental health stigma detection for the first time, leveraging clinical expert guidance to develop a fine-grained classification framework capable of systematically identifying and annotating both explicit and implicit stigmatizing language in model outputs. By incorporating a severity rating mechanism and an expanded multi-disorder stigma benchmark dataset, the proposed approach substantially outperforms traditional multiple-choice evaluation paradigms, effectively uncovering logical flaws in how models conceptualize psychological disorders and offering a novel pathway for diagnosing bias and guiding model refinement.

Abstract

While large language models (LLMs) are increasingly being explored for mental health applications, recent studies reveal that they can exhibit stigma toward individuals with psychological conditions. Existing evaluations of this stigma primarily rely on multiple-choice questions (MCQs), which fail to capture the biases embedded within the models' underlying logic. In this paper, we analyze the intermediate reasoning steps of LLMs to uncover hidden stigmatizing language and the internal rationales driving it. We leverage clinical expertise to categorize common patterns of stigmatizing language directed at individuals with psychological conditions and use this framework to identify and tag problematic statements in LLM reasoning. Furthermore, we rate the severity of these statements, distinguishing between overt prejudice and more subtle, less immediately harmful biases. To broaden the reasoning domain and capture a wider array of patterns, we also extend an existing mental health stigma benchmark by incorporating additional psychological conditions. Our findings demonstrate that evaluating model reasoning not only exposes substantially more stigma than traditional MCQ-based methods but it helps to identify the flaws in the LLMs' logic and their understanding of mental health conditions.

Problem

• LLM reasoning
• mental health stigma
• bias detection
• stigmatizing language
• psychological conditions

Innovation

• LLM reasoning analysis
• mental health stigma
• bias detection
• clinical framework
• reasoning benchmark extension