Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model

2025-02-02

AI Summary

This study addresses the low prediction reliability and poor uncertainty calibration of large language models (LLMs) in structuring Hebrew radiology reports—a high-stakes clinical NLP task. We propose the first agent-based, uncertainty-aware framework tailored for such critical medical applications. Methodologically, it integrates Bayesian prompt ensembling (BayesPE) with agent-driven decision modeling to generate five-level fine-grained confidence scores. It further incorporates uncertainty histogram analysis and a high-uncertainty sample filtering mechanism. Compared to conventional entropy-based uncertainty estimation, our framework significantly improves both calibration accuracy and interpretability. Evaluated on Llama 3.1 (8B-instruct) and HSMP-BERT, it achieves an F1 score of 0.4787 (post-filtering), a Cohen’s Kappa of 0.4258, and a recall of 0.6437, with clear separation between correct and incorrect predictions. This work establishes a novel paradigm for trustworthy AI in non-English clinical text processing.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Reliable extraction of structured data from radiology reports using Large Language Models (LLMs) remains challenging, especially for complex, non-English texts like Hebrew. This study introduces an agent-based uncertainty-aware approach to improve the trustworthiness of LLM predictions in medical applications. We analyzed 9,683 Hebrew radiology reports from Crohn's disease patients (from 2010 to 2023) across three medical centers. A subset of 512 reports was manually annotated for six gastrointestinal organs and 15 pathological findings, while the remaining reports were automatically annotated using HSMP-BERT. Structured data extraction was performed using Llama 3.1 (Llama 3-8b-instruct) with Bayesian Prompt Ensembles (BayesPE), which employed six semantically equivalent prompts to estimate uncertainty. An Agent-Based Decision Model integrated multiple prompt outputs into five confidence levels for calibrated uncertainty and was compared against three entropy-based models. Performance was evaluated using accuracy, F1 score, precision, recall, and Cohen's Kappa before and after filtering high-uncertainty cases. The agent-based model outperformed the baseline across all metrics, achieving an F1 score of 0.3967, recall of 0.6437, and Cohen's Kappa of 0.3006. After filtering high-uncertainty cases (greater than or equal to 0.5), the F1 score improved to 0.4787, and Kappa increased to 0.4258. Uncertainty histograms demonstrated clear separation between correct and incorrect predictions, with the agent-based model providing the most well-calibrated uncertainty estimates. By incorporating uncertainty-aware prompt ensembles and an agent-based decision model, this approach enhances the performance and reliability of LLMs in structured data extraction from radiology reports, offering a more interpretable and trustworthy solution for high-stakes medical applications.

Problem

• Improves LLM reliability in medical data extraction.
• Addresses uncertainty in non-English radiology reports.
• Enhances structured data extraction using agent-based models.

Innovation

• Agent-based uncertainty-aware approach
• Bayesian Prompt Ensembles (BayesPE)
• Agent-Based Decision Model