Unlocking Historical Clinical Trial Data with ALIGN: A Compositional Large Language Model System for Medical Coding

2024-11-20

arXiv.org

AI Summary

Historical clinical trial data suffer from poor cross-study interoperability due to the absence of standardized medical coding (e.g., ATC, MedDRA). To address this, we propose the first zero-shot, unsupervised, three-stage compositional LLM framework for automatic drug and diagnosis term coding: (1) multi-candidate generation, (2) self-consistency evaluation, and (3) uncertainty-driven human-in-the-loop decision-making—enabling trustworthy deployment with dynamic manual fallback. Leveraging GPT-4o-mini and GPT-4o, the framework integrates zero-shot prompting, confidence calibration, and epistemic uncertainty estimation. Experiments show MedDRA HLGT-level accuracy of 87–90%, and ATC Level 4 overall accuracy of 72–73% (reaching 86–89% for common drugs). With uncertainty-triggered human review, accuracy improves to 90% at a 30% fallback rate, incurring only $0.0007 per code.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

The reuse of historical clinical trial data has significant potential to accelerate medical research and drug development. However, interoperability challenges, particularly with missing medical codes, hinders effective data integration across studies. While Large Language Models (LLMs) offer a promising solution for automated coding without labeled data, current approaches face challenges on complex coding tasks. We introduce ALIGN, a novel compositional LLM-based system for automated, zero-shot medical coding. ALIGN follows a three-step process: (1) diverse candidate code generation; (2) self-evaluation of codes and (3) confidence scoring and uncertainty estimation enabling human deferral to ensure reliability. We evaluate ALIGN on harmonizing medication terms into Anatomical Therapeutic Chemical (ATC) and medical history terms into Medical Dictionary for Regulatory Activities (MedDRA) codes extracted from 22 immunology trials. ALIGN outperformed the LLM baselines, while also providing capabilities for trustworthy deployment. For MedDRA coding, ALIGN achieved high accuracy across all levels, matching RAG and excelling at the most specific levels (87-90% for HLGT). For ATC coding, ALIGN demonstrated superior performance, particularly at lower hierarchy levels (ATC Level 4), with 72-73% overall accuracy and 86-89% accuracy for common medications, outperforming baselines by 7-22%. ALIGN's uncertainty-based deferral improved accuracy by 17% to 90% accuracy with 30% deferral, notably enhancing performance on uncommon medications. ALIGN achieves this cost-efficiently at $0.0007 and $0.02 per code for GPT-4o-mini and GPT-4o, reducing barriers to clinical adoption. ALIGN advances automated medical coding for clinical trial data, contributing to enhanced data interoperability and reusability, positioning it as a promising tool to improve clinical research and accelerate drug development.

Problem

• Addresses interoperability challenges in historical clinical trial data reuse.
• Improves automated medical coding accuracy without labeled data.
• Enhances data integration and reusability for clinical research.

Innovation

• ALIGN: compositional LLM for zero-shot medical coding
• Three-step process: generation, self-evaluation, confidence scoring
• Uncertainty-based deferral improves accuracy and reliability