Residual Distribution Predictive Systems

2025-10-30

AI Summary

This work addresses the limitation of conformal prediction—its reliance on strictly valid conformity scores, which restricts model compatibility. We propose a conformity-score-free approach that models residuals as random variables and directly learns their distribution via split-sample or full-sample nonparametric statistics, enabling construction of prediction distributions with out-of-sample calibration guarantees. Our method applies to any point-estimate regression model: it treats model residuals as stochastic quantities and estimates their empirical distribution without assuming parametric forms or structural constraints (e.g., independence, symmetry, or homoscedasticity). We prove finite-sample cross-calibration—that is, marginal coverage guarantees hold uniformly across test samples. Empirically, our method achieves calibration performance comparable to standard conformal prediction while being inherently compatible with arbitrary regression models, including deep neural networks and ensemble methods. This significantly broadens the applicability and flexibility of calibrated prediction systems.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Conformal predictive systems are sets of predictive distributions with theoretical out-of-sample calibration guarantees. The calibration guarantees are typically that the set of predictions contains a forecast distribution whose prediction intervals exhibit the correct marginal coverage at all levels. Conformal predictive systems are constructed using conformity measures that quantify how well possible outcomes conform with historical data. However, alternative methods have been proposed to construct predictive systems with more appealing theoretical properties. We study an approach to construct predictive systems that we term Residual Distribution Predictive Systems. In the split conformal setting, this approach nests conformal predictive systems with a popular class of conformity measures, providing an alternative perspective on the classical approach. In the full conformal setting, the two approaches differ, and the new approach has the advantage that it does not rely on a conformity measure satisfying fairly stringent requirements to ensure that the predictive system is well-defined; it can readily be implemented alongside any point-valued regression method to yield predictive systems with out-of-sample calibration guarantees. The empirical performance of this approach is assessed using simulated data, where it is found to perform competitively with conformal predictive systems. However, the new approach offers considerable scope for implementation with alternative regression methods.

Problem

• Constructing predictive systems with theoretical calibration guarantees
• Developing Residual Distribution Predictive Systems as alternative method
• Ensuring well-defined predictive systems without stringent conformity measures

Innovation

• Residual Distribution Predictive Systems for predictive distributions
• Uses point-valued regression methods for calibration
• Ensures out-of-sample calibration guarantees without conformity measures