Adaptive Temperature Scaling with Conformal Prediction

2025-05-21

AI Summary

Existing conformal prediction methods output only prediction sets that cover the true label, without providing calibrated probability estimates for individual candidate labels within the set. This work proposes Input-Adaptive Temperature Scaling (IATS), the first method to assign single-label calibrated probabilities over conformal prediction sets while strictly preserving the prescribed coverage guarantee. Its core innovation is an input-dependent temperature parameter that dynamically calibrates model logits, ensuring marginal calibration of predicted probabilities over the adaptive set. Evaluated on multiple image classification benchmarks, IATS reduces Expected Calibration Error (ECE) by 30–50% compared to baselines, while exactly maintaining target coverage levels (e.g., 90%). By simultaneously achieving reliability (valid coverage) and discriminativeness (sharp, calibrated probabilities), IATS provides an interpretable and verifiable probabilistic interface for uncertainty quantification.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Conformal prediction enables the construction of high-coverage prediction sets for any pre-trained model, guaranteeing that the true label lies within the set with a specified probability. However, these sets do not provide probability estimates for individual labels, limiting their practical use. In this paper, we propose, to the best of our knowledge, the first method for assigning calibrated probabilities to elements of a conformal prediction set. Our approach frames this as an adaptive calibration problem, selecting an input-specific temperature parameter to match the desired coverage level. Experiments on several challenging image classification datasets demonstrate that our method maintains coverage guarantees while significantly reducing expected calibration error.

Problem

• Assign calibrated probabilities to conformal prediction sets
• Maintain coverage guarantees while reducing calibration error
• Adaptive calibration via input-specific temperature scaling

Innovation

• Adaptive temperature scaling for calibration
• Conformal prediction with calibrated probabilities
• Input-specific temperature parameter selection