Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text

2026-06-16

AI Summary

This study addresses the critical challenge that large language models (LLMs) often fail to accurately preserve expressions of diagnostic uncertainty in clinical text, potentially leading to erroneous clinical interpretations. The authors present the first systematic benchmark dataset comprising 1,200 clinical documents annotated with 9,184 five-level uncertainty labels. Leveraging clinical natural language processing and LLM evaluation methodologies, they comprehensively assess the performance of state-of-the-art models in retaining diagnostic uncertainty cues. Results reveal that current models achieve less than 50% accuracy in preserving original uncertainty signals and struggle to differentiate between adjacent uncertainty levels, highlighting a significant deficiency in this clinically vital dimension. This work establishes a foundational benchmark and provides clear directions for future model improvements.

Abstract

Large language models (LLMs) are increasingly used for clinical text tasks such as summarization and revision. While most studies evaluate the fluency and coherence of LLM-generated text, whether LLMs correctly preserve diagnostic uncertainty remains underexplored. In clinical practice, phrases such as ``possible pneumonia'' communicate the strength of available evidence and directly guide decisions about follow-up testing and treatment. Altering these uncertainty expressions can change the clinical meaning entirely. In this paper, we systematically evaluated this problem in two steps. First, we constructed a benchmark of 1,200 clinical documents with 9,184 uncertainty annotations across five levels. Second, we evaluated three LLMs on this benchmark. Our results show that (1) LLMs preserve the original uncertainty cues poorly, often less than half the time; (2) LLMs struggle with nuanced distinctions between adjacent levels. This work reveals a failure mode not captured by standard evaluation metrics and provides implications for the safe deployment of LLMs in clinical workflows.

Problem

• diagnostic uncertainty
• clinical text
• large language models
• uncertainty preservation
• clinical decision-making

Innovation

• diagnostic uncertainty
• clinical text generation
• LLM evaluation benchmark
• uncertainty preservation
• clinical safety