Adaptive Coverage Policies in Conformal Prediction

2025-10-05

AI Summary

Traditional conformal prediction employs a fixed coverage level, often yielding overly conservative or even empty prediction sets, and lacks sample-wise adaptivity. To address this, we propose an adaptive conformal prediction framework that dynamically adjusts the coverage level per sample based on estimated difficulty. Our method is the first to integrate e-values with posterior conformal inference, leveraging a neural network trained on a calibration set to learn data-dependent coverage policies. It employs leave-one-out training and variable-coverage optimization, ensuring strict marginal coverage guarantees while substantially reducing prediction set size. Experiments across diverse classification and regression tasks demonstrate consistent improvements over fixed-coverage baselines—achieving greater flexibility, computational efficiency, and theoretical rigor without compromising statistical validity.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Game Theory and Economic Paradigms: Adversarial Learning
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Traditional conformal prediction methods construct prediction sets such that the true label falls within the set with a user-specified coverage level. However, poorly chosen coverage levels can result in uninformative predictions, either producing overly conservative sets when the coverage level is too high, or empty sets when it is too low. Moreover, the fixed coverage level cannot adapt to the specific characteristics of each individual example, limiting the flexibility and efficiency of these methods. In this work, we leverage recent advances in e-values and post-hoc conformal inference, which allow the use of data-dependent coverage levels while maintaining valid statistical guarantees. We propose to optimize an adaptive coverage policy by training a neural network using a leave-one-out procedure on the calibration set, allowing the coverage level and the resulting prediction set size to vary with the difficulty of each individual example. We support our approach with theoretical coverage guarantees and demonstrate its practical benefits through a series of experiments.

Problem

• Adaptive coverage policies address fixed coverage limitations in conformal prediction
• Methods optimize prediction set sizes based on individual example difficulty
• Neural networks learn data-dependent coverage while maintaining statistical guarantees

Innovation

• Neural network optimizes adaptive coverage policy
• Leave-one-out training on calibration set
• Data-dependent coverage with theoretical guarantees