Task as Context Prompting for Accurate Medical Symptom Coding Using Large Language Models

2025-04-03

AI Summary

Low accuracy in symptom coding, difficulty identifying rare symptoms, and challenges posed by narrative diversity in unstructured clinical texts (e.g., vaccine safety reports) limit the performance of conventional multi-stage approaches. Method: We propose Task as Context (TACO), a unified prompting framework that jointly models fine-grained symptom extraction and MedDRA terminology linking. We introduce SYMPCODER—the first manually annotated dataset derived from the Vaccine Adverse Event Reporting System (VAERS)—and design a two-stage evaluation protocol. Leveraging context-aware, domain-knowledge-enhanced prompt engineering, we evaluate TACO on Llama2-chat, GPT-3.5/4 Turbo, and GPT-4o. Contribution/Results: TACO achieves significant improvements in symptom-to-MedDRA linking accuracy and mention fidelity. This work establishes a highly adaptable and scalable paradigm for standardized clinical text encoding.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Symbolic Representations

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Accurate medical symptom coding from unstructured clinical text, such as vaccine safety reports, is a critical task with applications in pharmacovigilance and safety monitoring. Symptom coding, as tailored in this study, involves identifying and linking nuanced symptom mentions to standardized vocabularies like MedDRA, differentiating it from broader medical coding tasks. Traditional approaches to this task, which treat symptom extraction and linking as independent workflows, often fail to handle the variability and complexity of clinical narratives, especially for rare cases. Recent advancements in Large Language Models (LLMs) offer new opportunities but face challenges in achieving consistent performance. To address these issues, we propose Task as Context (TACO) Prompting, a novel framework that unifies extraction and linking tasks by embedding task-specific context into LLM prompts. Our study also introduces SYMPCODER, a human-annotated dataset derived from Vaccine Adverse Event Reporting System (VAERS) reports, and a two-stage evaluation framework to comprehensively assess both symptom linking and mention fidelity. Our comprehensive evaluation of multiple LLMs, including Llama2-chat, Jackalope-7b, GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4o, demonstrates TACO's effectiveness in improving flexibility and accuracy for tailored tasks like symptom coding, paving the way for more specific coding tasks and advancing clinical text processing methodologies.

Problem

• Accurate symptom coding from clinical text for pharmacovigilance.
• Linking symptom mentions to MedDRA with unified extraction-linking.
• Improving LLM performance for rare and complex clinical cases.

Innovation

• Task as Context Prompting unifies extraction and linking
• SYMPCODER dataset enhances symptom coding accuracy
• Two-stage evaluation framework assesses symptom fidelity