Integrating Large Language Models with Human Expertise for Disease Detection in Electronic Health Records

2025-03-31

AI Summary

Manual annotation for multi-disease identification in electronic health records (EHRs) is labor-intensive and inefficient. Method: We propose the first prompt engineering framework integrating large language models (LLMs) with structured clinical knowledge—embedding clinical guidelines, ICD coding logic, and EHR text semantic parsing into domain-specific prompts to enable end-to-end automated identification of acute myocardial infarction (AMI), diabetes, and hypertension. Contribution/Results: Our approach jointly optimizes diagnostic accuracy and interpretability, achieving high sensitivity while significantly improving negative predictive value (NPV). Evaluated on 3,088 patients and 550,000 clinical notes, it attains sensitivities of 88% (AMI), 91% (diabetes), and 94% (hypertension), with corresponding NPVs surpassing those of traditional ICD-coded baselines across all three conditions. This substantially reduces reliance on manual annotation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Objective: Electronic health records (EHR) are widely available to complement administrative data-based disease surveillance and healthcare performance evaluation. Defining conditions from EHR is labour-intensive and requires extensive manual labelling of disease outcomes. This study developed an efficient strategy based on advanced large language models to identify multiple conditions from EHR clinical notes. Methods: We linked a cardiac registry cohort in 2015 with an EHR system in Alberta, Canada. We developed a pipeline that leveraged a generative large language model (LLM) to analyze, understand, and interpret EHR notes by prompts based on specific diagnosis, treatment management, and clinical guidelines. The pipeline was applied to detect acute myocardial infarction (AMI), diabetes, and hypertension. The performance was compared against clinician-validated diagnoses as the reference standard and widely adopted International Classification of Diseases (ICD) codes-based methods. Results: The study cohort accounted for 3,088 patients and 551,095 clinical notes. The prevalence was 55.4%, 27.7%, 65.9% and for AMI, diabetes, and hypertension, respectively. The performance of the LLM-based pipeline for detecting conditions varied: AMI had 88% sensitivity, 63% specificity, and 77% positive predictive value (PPV); diabetes had 91% sensitivity, 86% specificity, and 71% PPV; and hypertension had 94% sensitivity, 32% specificity, and 72% PPV. Compared with ICD codes, the LLM-based method demonstrated improved sensitivity and negative predictive value across all conditions. The monthly percentage trends from the detected cases by LLM and reference standard showed consistent patterns.

Problem

• Develop LLM-based pipeline for disease detection in EHR notes
• Improve accuracy compared to ICD codes for multiple conditions
• Automate labor-intensive manual labeling of disease outcomes

Innovation

• LLM analyzes EHR notes via diagnosis prompts
• Pipeline detects diseases using clinical guidelines
• LLM improves sensitivity over ICD codes