Multi-Label Classification with Generative AI Models in Healthcare: A Case Study of Suicidality and Risk Factors

2025-07-22

AI Summary

This study addresses limitations in identifying suicide-related risk factors—suicidal ideation, suicide attempt, suicide exposure, and non-suicidal self-injury—in psychiatric electronic health records (EHRs). We propose the first generative multi-label classification framework tailored to clinical text, departing from conventional binary classification. Our end-to-end pipeline integrates fine-tuned GPT-3.5 with GPT-4.5–guided prompting and introduces label-set–level evaluation metrics and multi-label confusion matrices to systematically characterize model error patterns and annotator conservatism bias. Experiments demonstrate that the fine-tuned GPT-3.5 achieves 0.94 partial-match accuracy and 0.91 macro-F1; GPT-4.5 significantly outperforms on rare label combinations, exhibiting superior robustness and class-balance capability. This work establishes an interpretable, rigorously evaluable generative AI paradigm for modeling complex, comorbid suicide risk in clinical NLP.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Suicide remains a pressing global health crisis, with over 720,000 deaths annually and millions more affected by suicide ideation (SI) and suicide attempts (SA). Early identification of suicidality-related factors (SrFs), including SI, SA, exposure to suicide (ES), and non-suicidal self-injury (NSSI), is critical for timely intervention. While prior studies have applied AI to detect SrFs in clinical notes, most treat suicidality as a binary classification task, overlooking the complexity of cooccurring risk factors. This study explores the use of generative large language models (LLMs), specifically GPT-3.5 and GPT-4.5, for multi-label classification (MLC) of SrFs from psychiatric electronic health records (EHRs). We present a novel end to end generative MLC pipeline and introduce advanced evaluation methods, including label set level metrics and a multilabel confusion matrix for error analysis. Finetuned GPT-3.5 achieved top performance with 0.94 partial match accuracy and 0.91 F1 score, while GPT-4.5 with guided prompting showed superior performance across label sets, including rare or minority label sets, indicating a more balanced and robust performance. Our findings reveal systematic error patterns, such as the conflation of SI and SA, and highlight the models tendency toward cautious over labeling. This work not only demonstrates the feasibility of using generative AI for complex clinical classification tasks but also provides a blueprint for structuring unstructured EHR data to support large scale clinical research and evidence based medicine.

Problem

• Classify multiple suicidality risk factors from EHRs
• Improve early detection of cooccurring suicide-related conditions
• Evaluate generative AI models for multi-label clinical tasks

Innovation

• Generative AI models for multi-label classification
• Fine-tuned GPT-3.5 achieves high accuracy
• Guided prompting enhances GPT-4.5 performance