Clinical Uncertainty Impacts Machine Learning Evaluations

2025-09-26

AI Summary

Clinical data annotation suffers from inter-annotator disagreement and heterogeneous label confidence, which conventional aggregation methods—such as majority voting—obscure by collapsing uncertainty into deterministic labels, thereby distorting model evaluation. To address this, we propose an uncertainty-aware evaluation paradigm that operates directly on raw multi-annotator labels. Our approach introduces a closed-form probabilistic metric with linear time complexity, explicitly modeling the underlying label distribution rather than discrete hard labels, and is compatible with diverse annotation protocols—including crowdsourcing and expert consensus. Empirical results demonstrate that our method substantially alters the ranking of binary classifiers, uncovering critical performance distinctions masked by majority voting. Our principal contributions are: (1) advancing medical AI evaluation from deterministic to probabilistic foundations; (2) advocating for transparent sharing of raw annotation data; and (3) providing an efficient, interpretable, and plug-and-play probabilistic evaluation tool.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Clinical dataset labels are rarely certain as annotators disagree and confidence is not uniform across cases. Typical aggregation procedures, such as majority voting, obscure this variability. In simple experiments on medical imaging benchmarks, accounting for the confidence in binary labels significantly impacts model rankings. We therefore argue that machine-learning evaluations should explicitly account for annotation uncertainty using probabilistic metrics that directly operate on distributions. These metrics can be applied independently of the annotations' generating process, whether modeled by simple counting, subjective confidence ratings, or probabilistic response models. They are also computationally lightweight, as closed-form expressions have linear-time implementations once examples are sorted by model score. We thus urge the community to release raw annotations for datasets and to adopt uncertainty-aware evaluation so that performance estimates may better reflect clinical data.

Problem

• Clinical labels lack certainty due to annotator disagreement
• Standard aggregation methods obscure label variability
• Machine learning evaluations should incorporate annotation uncertainty

Innovation

• Using probabilistic metrics for annotation uncertainty
• Applying metrics independent of annotation generating process
• Implementing computationally lightweight linear-time evaluations