Selecting Informative Conformal Prediction Sets with an Optimized FCR-Controlled Approach

2026-05-21

AI Summary

This work addresses the selection bias inherent in selective inference when inference is performed only on high-informativeness prediction sets, which can compromise false coverage rate (FCR) control. Under an idealized setting, the authors derive an oracle strategy that maximizes statistical power while maintaining FCR guarantees. They then develop a finite-sample calibration mechanism grounded in conformal prediction and probability calibration to adapt this optimal strategy to practical settings. The resulting procedure rigorously controls FCR while achieving substantially higher power than existing methods. Empirical evaluations on both synthetic and real-world classification datasets demonstrate consistent superiority over current approaches in terms of statistical efficiency without sacrificing FCR control.

Abstract

Conformal methods provide prediction sets for outcomes with confidence guarantees. We study their use in a selective inference setting, where inference is performed only when the prediction set is informative. The analyst may consider as informative, for example, cases with prediction sets that are sufficiently small, exclude null values, or satisfy other appropriate monotone constraints. Because inference is typically restricted to informative cases in practical applications, accounting for the resulting selection bias is crucial to maintaining false coverage rate (FCR) control. A general framework for constructing such informative conformal prediction sets while controlling the FCR on the selected sample was suggested in Gazin et al. (2025). In this work we focus on oracle-guided procedures. We derive the optimal decision policy under a suitable power objective in the oracle setting where the probability of belonging to each prediction set can be computed. In practice, of course, only estimated probabilities are available. We therefore introduce a calibration procedure that adjusts the oracle policy to maintain finite sample FCR control. We show that this approach can achieve substantially higher power than available alternatives. We demonstrate the effectiveness of our new methods for classification outcomes on both real and simulated data.

Problem

• Conformal Prediction
• Selective Inference
• False Coverage Rate
• Informative Prediction Sets
• Selection Bias

Innovation

• Conformal Prediction
• False Coverage Rate (FCR)
• Selective Inference
• Oracle Policy
• Calibration Procedure