Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models

2025-02-08

AI Summary

In safety-critical applications, the opaque decision-making of large language and vision-language models hinders rigorous risk assessment; existing static-threshold conformal prediction methods fail to jointly optimize accuracy, coverage, and informativeness. To address this, we propose a learnable conformal rejection policy that— for the first time—models the conformal threshold as a dynamic action within a deep reinforcement learning framework, enabling multi-objective optimization of uncertainty quantification metrics. Our approach integrates conformal prediction, uncertainty-driven selective generation, and a policy-based rejection mechanism. Evaluated across multiple models and datasets, it guarantees 90% statistical coverage while improving accuracy by 3.2%, increasing hallucination detection AUROC by 22.19%, and reducing calibration error by 70–85%. The method significantly improves the accuracy–coverage trade-off, offering a principled, adaptive solution for trustworthy AI deployment in high-stakes settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Large-scale security measurements

Abstract

Large Language and Vision-Language Models (LLMs/VLMs) are increasingly used in safety-critical applications, yet their opaque decision-making complicates risk assessment and reliability. Uncertainty quantification (UQ) helps assess prediction confidence and enables abstention when uncertainty is high. Conformal prediction (CP), a leading UQ method, provides statistical guarantees but relies on static thresholds, which fail to adapt to task complexity and evolving data distributions, leading to suboptimal trade-offs in accuracy, coverage, and informativeness. To address this, we propose learnable conformal abstention, integrating reinforcement learning (RL) with CP to optimize abstention thresholds dynamically. By treating CP thresholds as adaptive actions, our approach balances multiple objectives, minimizing prediction set size while maintaining reliable coverage. Extensive evaluations across diverse LLM/VLM benchmarks show our method outperforms Least Ambiguous Classifiers (LAC) and Adaptive Prediction Sets (APS), improving accuracy by up to 3.2%, boosting AUROC for hallucination detection by 22.19%, enhancing uncertainty-guided selective generation (AUARC) by 21.17%, and reducing calibration error by 70%-85%. These improvements hold across multiple models and datasets while consistently meeting the 90% coverage target, establishing our approach as a more effective and flexible solution for reliable decision-making in safety-critical applications. The code is available at: {https://github.com/sinatayebati/vlm-uncertainty}.

Problem

• Dynamic abstention thresholds in conformal prediction
• Adaptive risk management in LLMs/VLMs
• Optimizing accuracy, coverage, and informativeness trade-offs

Innovation

• Reinforcement learning optimizes abstention thresholds
• Dynamic conformal prediction balances objectives
• Enhances accuracy and uncertainty detection