🤖 AI Summary
This work addresses the challenge that large language models often produce fluent yet incorrect answers in question answering, while existing uncertainty scoring methods struggle to reliably distinguish correct from incorrect responses and fail to statistically control error rates with fixed thresholds. To overcome these limitations, the authors propose A-CRC-QA, a novel framework that introduces asymptotic risk calibration into selective question answering for the first time. Inspired by conformal risk control, their approach employs a monotonic empirical risk calibration procedure that reformulates error control as a linear expectation constraint. The method is training-free, model-agnostic, compatible with diverse uncertainty estimators, and applicable to both open-ended and closed-form QA tasks. Experiments on CoQA and MedMCQA demonstrate that A-CRC-QA achieves a significantly better trade-off between answer retention rate and reliability compared to uncalibrated baselines and confidence-bound approaches.
📝 Abstract
Large language models (LLMs) may generate fluent but incorrect answers, making uncertainty quantification important for reliable question answering. However, heuristic uncertainty scores cannot perfectly distinguish correct predictions from incorrect ones, and directly applying a fixed uncertainty threshold provides no statistical control over the error rate among accepted answers. To address this limitation, we propose A-CRC-QA, a post-hoc calibration framework for uncertainty-aware selective question answering. The proposed method reformulates selection-conditioned error control as a linear expectation constraint and applies a monotonized empirical-risk calibration procedure inspired by conformal risk control. Since the resulting instance-wise loss is generally non-monotone with respect to the acceptance threshold, our framework targets asymptotic rather than finite-sample risk control. A-CRC-QA is model-agnostic, requires no additional training, and can be combined with different uncertainty estimators. Experiments on CoQA and MedMCQA demonstrate its applicability to both open-ended and closed-ended question answering, achieving a favorable trade-off between accepted-answer reliability and answer retention compared with uncalibrated and confidence-bound-based baselines.