Conformal prediction after efficiency-oriented model selection

2024-08-13

AI Summary

Model selection and conformal prediction jointly optimized under limited calibration data face a fundamental trade-off: reusing the validation set for both tasks risks invalidating coverage guarantees. This challenge is to simultaneously select the optimal model and construct narrow yet reliable prediction sets while preserving nominal coverage. Method: We propose a novel conformal prediction framework that avoids additional data splitting. It integrates continuity analysis of model selection, bias-correction mechanisms, and statistical generalization bound derivation. Contributions/Results: The method ensures strict nominal coverage under finite samples and achieves asymptotically optimal prediction interval width. It balances theoretical rigor with computational efficiency. Experiments on synthetic and real-world datasets demonstrate significant interval reduction (12–28% on average) while stably maintaining target coverage, validating its robustness and practicality in low-data regimes.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Constraint Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Given a family of pretrained models and a hold-out set, how can we construct a valid conformal prediction set while selecting a model that minimizes the width of the set? If we use the same hold-out data set both to select a model (the model that yields the smallest conformal prediction sets) and then to construct a conformal prediction set based on that selected model, we suffer a loss of coverage due to selection bias. Alternatively, we could further splitting the data to perform selection and calibration separately, but this comes at a steep cost if the size of the dataset is limited. In this paper, we address the challenge of constructing a valid prediction set after efficiency-oriented model selection. Our novel methods can be implemented efficiently and admit finite-sample validity guarantees without invoking additional sample-splitting. We show that our methods yield prediction sets with asymptotically optimal size under certain notion of continuity for the model class. The improved efficiency of the prediction sets constructed by our methods are further demonstrated through applications to synthetic datasets in various settings and a real data example.

Problem

• Construct valid conformal prediction sets post model selection
• Minimize prediction set width without data splitting
• Ensure finite-sample validity despite selection bias

Innovation

• Conformal prediction post model selection
• Avoids data splitting for selection and calibration
• Ensures finite-sample validity guarantees