SConU: Selective Conformal Uncertainty in Large Language Models

2025-04-19

AI Summary

Large language models (LLMs) deployed in high-stakes scenarios often violate the exchangeability assumption underlying conformal prediction, leading to uncontrolled miscoverage and operationally infeasible prediction sets. Method: We propose a novel selective conformal prediction framework that—uniquely—introduces two types of conformal p-values for significance testing, enabling automatic detection and risk-controlled removal of samples with anomalous uncertainty distributions. Integrated with split conformal prediction, our approach jointly performs calibration-set distribution consistency assessment and approximate conditional coverage optimization. Contribution/Results: The method achieves near-target coverage both within-domain and cross-domain, substantially reduces miscoverage, and improves prediction set compactness and operational utility. Experiments on high-risk question-answering tasks demonstrate its ability to simultaneously enforce strict risk control and enhance predictive efficiency.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

As large language models are increasingly utilized in real-world applications, guarantees of task-specific metrics are essential for their reliable deployment. Previous studies have introduced various criteria of conformal uncertainty grounded in split conformal prediction, which offer user-specified correctness coverage. However, existing frameworks often fail to identify uncertainty data outliers that violate the exchangeability assumption, leading to unbounded miscoverage rates and unactionable prediction sets. In this paper, we propose a novel approach termed Selective Conformal Uncertainty (SConU), which, for the first time, implements significance tests, by developing two conformal p-values that are instrumental in determining whether a given sample deviates from the uncertainty distribution of the calibration set at a specific manageable risk level. Our approach not only facilitates rigorous management of miscoverage rates across both single-domain and interdisciplinary contexts, but also enhances the efficiency of predictions. Furthermore, we comprehensively analyze the components of the conformal procedures, aiming to approximate conditional coverage, particularly in high-stakes question-answering tasks.

Problem

• Ensuring task-specific reliability in large language model deployments
• Identifying uncertainty outliers violating exchangeability in conformal prediction
• Achieving conditional coverage in high-stakes question-answering tasks

Innovation

• Selective Conformal Uncertainty (SConU) for LLMs
• Uses two conformal p-values for outlier detection
• Manages miscoverage rates in single and interdisciplinary contexts