Null-Calibrated Conformal Selection via Target-Membership Scores

2026-06-21

AI Summary

This study addresses the misalignment between conventional prediction-oriented scoring and selection objectives in complex target regions—such as intervals, variance-driven sets, multimodal distributions, or multi-condition scenarios. To resolve this, the authors propose using target membership probability as a nonconformity score to directly rank binary selection events, combined with a null-calibrated conformal selection (NCCS) procedure that leverages non-target calibration samples to produce finite-sample valid p-values. This approach is the first to explicitly distinguish prediction from selection tasks and formalizes a target membership scoring principle. While maintaining comparable performance under mean-monotonic targets, it substantially improves selection efficacy in variance-driven and other complex settings. Moreover, in rare-target regimes, NCCS effectively mitigates the anti-conservatism of empirical FDP thresholds, achieving both high power and rigorous control of the false discovery rate in finite samples.

Abstract

Conformal selection aims to identify test candidates whose unknown responses fall in a target region while controlling the false discovery rate. Existing methods often inherit prediction-oriented nonconformity scores, such as residual or clipped residual scores, from conformal prediction. We argue that the natural score for selection is instead the target-membership probability. This score directly addresses the binary event being selected, and any monotone transform of it gives the Neyman--Pearson oracle ranking at a fixed null selection level. This distinction is irrelevant for mean-monotone targets, where conventional scores induce essentially the same ranking, but becomes important for interval-valued, variance-driven, multimodal, or multi-condition targets, where prediction-oriented scores can be misaligned with selection power. We study membership-score-based conformal selection and isolate one conformal calibration route, Null-Calibrated Conformal Selection (NCCS), which ranks test scores against confirmed non-target calibration examples. Under null exchangeability, NCCS yields finite-sample valid null p-values, which can be combined with BY under arbitrary dependence or with BH under standard positive-dependence conditions. Experiments support the score principle: membership scores match conventional scores on mean-monotone targets, substantially improve over mean-score selection on variance-driven targets, and, when calibrated by NCCS, trade power for finite-sample null validity in rare-target regimes where direct empirical-FDP thresholding can be anti-conservative.

Problem

• conformal selection
• false discovery rate
• target-membership probability
• nonconformity scores
• null calibration

Innovation

• target-membership score
• conformal selection
• null calibration
• finite-sample validity
• false discovery rate