Automating Adjudication of Cardiovascular Events Using Large Language Models

2025-03-21

AI Summary

Cardiovascular event adjudication represents a critical bottleneck in clinical trials, as conventional manual approaches are time-intensive, highly subjective, and exhibit poor reproducibility. To address this, we propose the first two-stage large language model (LLM) framework specifically designed for cardiovascular event adjudication. In Stage I, domain-adapted named entity recognition and relation extraction precisely extract event-related information from unstructured clinical text (F1 = 0.82). In Stage II, guideline-compliant reasoning—integrating Cardiovascular Endpoint Committee (CEC) standards with Tree-of-Thoughts prompting—enables interpretable, regulatory-aligned automated adjudication (accuracy = 0.68). We introduce CLEART, a novel evaluation metric tailored to adjudication quality, and demonstrate significant reductions in adjudication turnaround time, inter-adjudicator variability, and enhanced auditability. This work establishes a new paradigm for AI-augmented endpoint determination in cardiovascular clinical research.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Cardiovascular events, such as heart attacks and strokes, remain a leading cause of mortality globally, necessitating meticulous monitoring and adjudication in clinical trials. This process, traditionally performed manually by clinical experts, is time-consuming, resource-intensive, and prone to inter-reviewer variability, potentially introducing bias and hindering trial progress. This study addresses these critical limitations by presenting a novel framework for automating the adjudication of cardiovascular events in clinical trials using Large Language Models (LLMs). We developed a two-stage approach: first, employing an LLM-based pipeline for event information extraction from unstructured clinical data and second, using an LLM-based adjudication process guided by a Tree of Thoughts approach and clinical endpoint committee (CEC) guidelines. Using cardiovascular event-specific clinical trial data, the framework achieved an F1-score of 0.82 for event extraction and an accuracy of 0.68 for adjudication. Furthermore, we introduce the CLEART score, a novel, automated metric specifically designed for evaluating the quality of AI-generated clinical reasoning in adjudicating cardiovascular events. This approach demonstrates significant potential for substantially reducing adjudication time and costs while maintaining high-quality, consistent, and auditable outcomes in clinical trials. The reduced variability and enhanced standardization also allow for faster identification and mitigation of risks associated with cardiovascular therapies.

Problem

• Automating cardiovascular event adjudication in clinical trials using LLMs
• Reducing manual effort and variability in event adjudication processes
• Improving speed and cost-efficiency while maintaining adjudication quality

Innovation

• LLM-based pipeline for event extraction
• Tree of Thoughts guided adjudication process
• CLEART score for AI reasoning evaluation