Robust Decision Making with Partially Calibrated Forecasts

2025-10-27

AI Summary

This paper studies robust decision-making under high-dimensional predictions subject to partial calibration—specifically, when predictions satisfy “decision calibration,” a weaker requirement than classical calibration. It addresses how a conservative decision-maker can maximize worst-case expected utility under this relaxed condition. Methodologically, the paper develops a duality-based framework for deriving minimax-optimal decision rules and proves that classical strategies retain minimax optimality under decision calibration; for even weaker calibration levels, it constructs efficiently computable robust decision mappings. Crucially, the approach dispenses with the stringent assumption of full calibration, thereby substantially broadening the theoretical applicability of calibration in complex predictive tasks. Empirical evaluation on squared-error regression models demonstrates that the proposed rules achieve an exceptional trade-off between utility robustness and computational tractability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Decision/Utility Theory
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Calibration has emerged as a foundational goal in ``trustworthy machine learning'', in part because of its strong decision theoretic semantics. Independent of the underlying distribution, and independent of the decision maker's utility function, calibration promises that amongst all policies mapping predictions to actions, the uniformly best policy is the one that ``trusts the predictions'' and acts as if they were correct. But this is true only of emph{fully calibrated} forecasts, which are tractable to guarantee only for very low dimensional prediction problems. For higher dimensional prediction problems (e.g. when outcomes are multiclass), weaker forms of calibration have been studied that lack these decision theoretic properties. In this paper we study how a conservative decision maker should map predictions endowed with these weaker (``partial'') calibration guarantees to actions, in a way that is robust in a minimax sense: i.e. to maximize their expected utility in the worst case over distributions consistent with the calibration guarantees. We characterize their minimax optimal decision rule via a duality argument, and show that surprisingly, ``trusting the predictions and acting accordingly'' is recovered in this minimax sense by emph{decision calibration} (and any strictly stronger notion of calibration), a substantially weaker and more tractable condition than full calibration. For calibration guarantees that fall short of decision calibration, the minimax optimal decision rule is still efficiently computable, and we provide an empirical evaluation of a natural one that applies to any regression model solved to optimize squared error.

Problem

• Robust decision making with partially calibrated forecasts
• Minimax optimal decision rules for weak calibration
• Decision calibration recovers trust in predictions

Innovation

• Minimax robust decision making with partial calibration
• Duality argument characterizes optimal decision rule
• Decision calibration enables trust in predictions efficiently