Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports

2026-06-17

AI Summary

Existing evaluation metrics for radiology reports lack clinical grounding and struggle to distinguish clinically critical errors from benign phrasing variations. This work proposes the first LLM-based evaluation framework explicitly focused on clinical significance boundaries. Leveraging the ReEvalMed benchmark, the authors synthesize 4,000 paired reports using Qwen3-8B and MedGemma-4B to develop a lightweight, interpretable metric and compare single-pass versus two-pass inference strategies. Experimental results demonstrate that the proposed metric surpasses 32B-scale medical LLMs in assessing clinical significance and achieves performance comparable to proprietary models. Furthermore, the study reveals that two-pass inference does not consistently enhance overall effectiveness but instead entails a trade-off between discriminative power and robustness, while exhibiting pervasive judgment bias.

Abstract

Reliable evaluation of generated radiology reports requires strict clinical accuracy, as omitted critical findings or mischaracterized radiographic observations can directly affect patient care. Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar. Although Large Language Models (LLMs) possess rich medical knowledge, they likewise struggle to draw a reliable boundary between clinically significant errors and harmless variation. We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance from detecting true clinical errors ("Discrimination") and tolerating insignificant variations ("Robustness"). Across 8 LLM evaluators under one-pass and two-pass settings, we identify a widespread discrimination bias: models effectively detect errors but also over-penalize harmless rephrasings. To mitigate this, we synthesize 4k report pairs and train lightweight interpretable metrics on Qwen3-8B and MedGemma-4B. Our trained metric sharpens the clinical significance boundary, surpassing 32B-scale medical LLMs and remaining competitive with proprietary models. Crucially, the more costly two-pass setting fails to consistently improve overall performance and mainly trades discrimination for robustness. These findings suggest one-pass trained metrics as the practical choice for cost-sensitive deployment, with two-pass inference reserved for settings where D-R balance is critical. We will release the dataset and metric.

Problem

• clinical significance
• radiology report evaluation
• LLM-based metrics
• discrimination bias
• medical accuracy

Innovation

• clinical significance
• LLM-based metrics
• radiology report evaluation
• discrimination-robustness trade-off
• lightweight interpretable metrics