The Promise of Large Language Models in Digital Health: Evidence from Sentiment Analysis in Online Health Communities

2025-08-19

AI Summary

Digital health analytics faces challenges including emotionally complex patient-generated text, dense clinical terminology, scarce labeled data, and stringent privacy constraints—limiting the accuracy and regulatory compliance of conventional machine learning approaches. To address these, we propose a fine-tuning-free large language model (LLM)-driven sentiment analysis framework: structured medical codebooks guide in-context learning for models including GPT, DeepSeek, and LLaMA 3.1; results are validated via ensemble comparison with BioBERT variants and lexicon-based methods. Evaluated on 400 expert-annotated posts from online health communities, our method achieves statistical agreement with clinical experts (p > 0.05) and significantly outperforms traditional baselines. Crucially, it eliminates reliance on sensitive patient data for model training or manual annotation, enabling real-time, accurate, interpretable, and privacy-preserving sentiment analysis.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Brain-Sensing and Analysis

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Digital health analytics face critical challenges nowadays. The sophisticated analysis of patient-generated health content, which contains complex emotional and medical contexts, requires scarce domain expertise, while traditional ML approaches are constrained by data shortage and privacy limitations in healthcare settings. Online Health Communities (OHCs) exemplify these challenges with mixed-sentiment posts, clinical terminology, and implicit emotional expressions that demand specialised knowledge for accurate Sentiment Analysis (SA). To address these challenges, this study explores how Large Language Models (LLMs) can integrate expert knowledge through in-context learning for SA, providing a scalable solution for sophisticated health data analysis. Specifically, we develop a structured codebook that systematically encodes expert interpretation guidelines, enabling LLMs to apply domain-specific knowledge through targeted prompting rather than extensive training. Six GPT models validated alongside DeepSeek and LLaMA 3.1 are compared with pre-trained language models (BioBERT variants) and lexicon-based methods, using 400 expert-annotated posts from two OHCs. LLMs achieve superior performance while demonstrating expert-level agreement. This high agreement, with no statistically significant difference from inter-expert agreement levels, suggests knowledge integration beyond surface-level pattern recognition. The consistent performance across diverse LLM models, supported by in-context learning, offers a promising solution for digital health analytics. This approach addresses the critical challenge of expert knowledge shortage in digital health research, enabling real-time, expert-quality analysis for patient monitoring, intervention assessment, and evidence-based health strategies.

Problem

• Analyzing complex emotional contexts in patient-generated health content
• Overcoming data scarcity and privacy limitations in healthcare analytics
• Integrating expert knowledge for accurate sentiment analysis in OHCs

Innovation

• LLMs integrate expert knowledge via in-context learning
• Structured codebook enables domain-specific prompting without training
• LLMs achieve expert-level agreement in sentiment analysis