Conformal Prediction and Human Decision Making

2025-03-12

AI Summary

In high-stakes domains (e.g., healthcare, finance), model uncertainty remains poorly aligned with human decision-making needs. Method: We systematically model conformal prediction sets as decision-support tools—introducing the first evaluation framework for conformal prediction tailored to human decision-makers, integrating decision-theoretic modeling, cognitive behavioral experiments, and human-AI collaboration analysis. Contribution/Results: We identify the cognitive alignment mechanisms—and tensions—between conformal sets and threshold-sensitive, goal-directed human strategies. Crucially, we establish necessary conditions for their superiority over alternatives (e.g., point estimates + confidence scores): namely, tasks requiring explicit misclassification cost control, set-based judgments, or well-defined action thresholds. Our work provides theoretical boundaries, empirical evidence, and actionable design principles for trustworthy AI-assisted decision support.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Reasoning under Uncertainty: Decision/Utility Theory
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Methods to quantify uncertainty in predictions from arbitrary models are in demand in high-stakes domains like medicine and finance. Conformal prediction has emerged as a popular method for producing a set of predictions with specified average coverage, in place of a single prediction and confidence value. However, the value of conformal prediction sets to assist human decisions remains elusive due to the murky relationship between coverage guarantees and decision makers' goals and strategies. How should we think about conformal prediction sets as a form of decision support? Under what conditions do we expect the support they provide to be superior versus inferior to that of alternative presentations of predictive uncertainty? We outline a decision theoretic framework for evaluating predictive uncertainty as informative signals, then contrast what can be said within this framework about idealized use of calibrated probabilities versus conformal prediction sets. Informed by prior empirical results and theories of human decisions under uncertainty, we formalize a set of possible strategies by which a decision maker might use a prediction set. We identify ways in which conformal prediction sets and posthoc predictive uncertainty quantification more broadly are in tension with common goals and needs in human-AI decision making. We give recommendations for future research in predictive uncertainty quantification to support human decision makers.

Problem

• Quantify uncertainty in predictions for high-stakes domains
• Evaluate conformal prediction sets as decision support tools
• Address tension between prediction sets and human decision goals

Innovation

• Conformal prediction for uncertainty quantification
• Decision theoretic framework evaluation
• Strategies for human-AI decision making