An Information Theoretic Perspective on Conformal Prediction

2024-05-03

Neural Information Processing Systems

AI Summary

This work addresses two fundamental limitations of conformal prediction (CP): ambiguous semantic interpretation of uncertainty and rigid offline calibration. First, it establishes—novelty—theoretical connections between CP and conditional entropy from an information-theoretic perspective, rigorously characterizing the inherent uncertainty in input–output relationships. Second, it derives three computable upper bounds on conditional entropy using information inequalities—including the data processing inequality and Fano-type bounds—and designs a differentiable conformal training objective with side-information embedding, thereby overcoming traditional offline calibration constraints. The method integrates differentiable quantile regression, parameterized conformity scores, and a federated learning–compatible framework. Extensive experiments under both centralized and federated settings demonstrate significant reductions in average prediction set size (i.e., inefficiency), empirically validating the tightness of the proposed theoretical bounds and the generalization advantage of the new training paradigm.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Security and Privacy: Large-scale security measurements

Abstract

Conformal Prediction (CP) is a distribution-free uncertainty estimation framework that constructs prediction sets guaranteed to contain the true answer with a user-specified probability. Intuitively, the size of the prediction set encodes a general notion of uncertainty, with larger sets associated with higher degrees of uncertainty. In this work, we leverage information theory to connect conformal prediction to other notions of uncertainty. More precisely, we prove three different ways to upper bound the intrinsic uncertainty, as described by the conditional entropy of the target variable given the inputs, by combining CP with information theoretical inequalities. Moreover, we demonstrate two direct and useful applications of such connection between conformal prediction and information theory: (i) more principled and effective conformal training objectives that generalize previous approaches and enable end-to-end training of machine learning models from scratch, and (ii) a natural mechanism to incorporate side information into conformal prediction. We empirically validate both applications in centralized and federated learning settings, showing our theoretical results translate to lower inefficiency (average prediction set size) for popular CP methods.

Problem

• Connect conformal prediction to information theory
• Upper bound intrinsic uncertainty using CP
• Apply CP in machine learning training objectives

Innovation

• Leverages information theory
• Connects CP to uncertainty
• Enables end-to-end training