Singleton-Optimized Conformal Prediction

2025-09-28

AI Summary

Traditional conformal prediction guarantees marginal coverage but often yields overly large prediction sets, undermining decision-making utility. Existing methods typically optimize average set size rather than maximizing the probability of singleton predictions—i.e., unique, decisive outputs. This work proposes a novel nonconformity scoring method explicitly designed to maximize the frequency of singleton predictions. Leveraging the geometric structure inherent in K-class classification and a split-conformal framework, we devise an O(K)-time scoring function that directly minimizes the probability of non-singleton prediction sets. Efficient optimization is achieved via geometric reformulation. Experiments on image classification and large-language-model multiple-choice question answering demonstrate up to a 20% absolute increase in singleton prediction frequency, with negligible change in average prediction set size. This substantially enhances prediction definiteness and practical applicability while preserving rigorous coverage guarantees.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Constraint Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Conformal prediction can be used to construct prediction sets that cover the true outcome with a desired probability, but can sometimes lead to large prediction sets that are costly in practice. The most useful outcome is a singleton prediction-an unambiguous decision-yet existing efficiency-oriented methods primarily optimize average set size. Motivated by this, we propose a new nonconformity score that aims to minimize the probability of producing non-singleton sets. Starting from a non-convex constrained optimization problem as a motivation, we provide a geometric reformulation and associated algorithm for computing the nonconformity score and associated split conformal prediction sets in O(K) time for K-class problems. Using this score in split conformal prediction leads to our proposed Singleton-Optimized Conformal Prediction (SOCOP) method. We evaluate our method in experiments on image classification and LLM multiple-choice question-answering, comparing with standard nonconformity scores such as the (negative) label probability estimates and their cumulative distribution function; both of which are motivated by optimizing length. The results show that SOCOP increases singleton frequency (sometimes by over 20%) compared to the above scores, with minimal impact on average set size.

Problem

• Optimizes conformal prediction to increase singleton prediction frequency
• Minimizes probability of non-singleton sets while maintaining coverage
• Improves decision clarity in classification tasks with efficient computation

Innovation

• Optimizes nonconformity score to minimize non-singleton sets
• Provides geometric reformulation and O(K) time algorithm
• Increases singleton frequency with minimal average size impact