Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of Counterfactual Outcomes: A Case Study in Neurological Prognostication

2026-08-12

AI Summary

This study addresses label uncertainty in clinical prediction models when outcomes are unobservable due to therapeutic interventions, a challenge particularly acute in assessing neurological prognosis after cardiac arrest. The authors propose a novel framework that explicitly distinguishes patients with observed outcomes from those whose labels rely on expert-annotated counterfactuals, jointly modeling and evaluating both types of labels. They design a lightweight neural network with tunable contributions from each label type and reveal, for the first time, an inherent trade-off between accuracy and uncertainty calibration that conventional metrics like AUROC obscure: models achieving similar AUROC on certain cases exhibit markedly different Brier scores and probability estimates on uncertain cases, and improving fit to uncertain labels often comes at the cost of reduced accuracy on certain ones.

Abstract

Clinical prediction models are often developed as if the outcome of interest were cleanly observed for every patient. This assumption fails when treatment decisions make the clinically relevant outcome permanently unobservable. As a case study of this problem, we consider post-cardiac-arrest neurological prognostication using a cohort of 2,497 patients, including 1,429 patients whose outcomes were rendered indeterminate by treatment decisions. These patients with indeterminate outcomes were reviewed by independent clinical experts, who provided their guesses of counterfactual outcomes about what would have happened to the patients. We refer to these patients as uncertain cases. We also have patients for whom we observe their clinically relevant outcomes; we refer to these patients as certain cases. We propose a framework for evaluating prediction models that explicitly splits the evaluation between certain and uncertain cases. Here, we cannot easily evaluate both types of cases in a uniform manner as the available target labels differ. We then propose a simple prediction model that uses target labels from both certain and uncertain cases in a manner that allows us to trade off between them. Across the proposed neural model and a collection of tabular baselines, models with similar certain-case AUROC can nevertheless differ substantially in both certain-case Brier score and their probability estimates for uncertain cases. Improving alignment with target labels of uncertain cases for our proposed model generally comes at the cost of worse accuracy on certain cases, highlighting an explicit tradeoff that standard evaluation conceals. These results show that when treatment decisions determine whether clinically meaningful outcomes remain observable, conventional evaluation metrics can miss important failure modes in the very patients for whom prognostic support matters most.

Problem

• treatment-induced label indeterminacy
• counterfactual outcomes
• clinical prediction
• neurological prognostication
• outcome observability

Innovation

• treatment-induced label indeterminacy
• counterfactual outcome annotation
• uncertain case modeling
• split evaluation framework
• prognostic tradeoff