Optimizing Large Language Models for Detecting Symptoms of Comorbid Depression or Anxiety in Chronic Diseases: Insights from Patient Messages

2025-03-14

AI Summary

Early identification of comorbid depression/anxiety in diabetic patients remains clinically challenging, necessitating reliable, low-resource NLP solutions for real-world clinical messaging. Method: We systematically evaluate multiple large language models (LLMs) on zero-shot and few-shot mental health symptom detection using authentic, de-identified clinical safety messages. We propose a lightweight adaptation framework integrating engineered prompting, systematic role-based modeling, temperature tuning, and semantic parsing of the PHQ-4 scale to enable fine-grained symptom classification. Contribution/Results: Llama 3.1 405B achieves 93% F1 score and accuracy; three leading LLMs all exceed 90% in both metrics—demonstrating high reliability for binary classification and complex psychometric scale interpretation. This is the first study to rigorously validate LLMs for mental health screening in authentic clinical message contexts. Our framework establishes a reproducible, resource-efficient adaptation paradigm for low-resource medical NLP applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Patients with diabetes are at increased risk of comorbid depression or anxiety, complicating their management. This study evaluated the performance of large language models (LLMs) in detecting these symptoms from secure patient messages. We applied multiple approaches, including engineered prompts, systemic persona, temperature adjustments, and zero-shot and few-shot learning, to identify the best-performing model and enhance performance. Three out of five LLMs demonstrated excellent performance (over 90% of F-1 and accuracy), with Llama 3.1 405B achieving 93% in both F-1 and accuracy using a zero-shot approach. While LLMs showed promise in binary classification and handling complex metrics like Patient Health Questionnaire-4, inconsistencies in challenging cases warrant further real-life assessment. The findings highlight the potential of LLMs to assist in timely screening and referrals, providing valuable empirical knowledge for real-world triage systems that could improve mental health care for patients with chronic diseases.

Problem

• Detecting depression or anxiety symptoms in chronic disease patients
• Optimizing large language models for symptom detection
• Improving mental health care through timely screening and referrals

Innovation

• Engineered prompts and systemic persona used
• Zero-shot and few-shot learning applied
• Temperature adjustments enhanced model performance