🤖 AI Summary
Large language models (LLMs) frequently exhibit hallucination and overconfidence in multiple-choice question answering (MCQA), undermining their reliability. Method: We propose a black-box, model-agnostic uncertainty quantification method grounded in answer-frequency statistics—replacing logit-based probabilities with empirical frequencies from multiple independent samplings to construct a distribution-free conformal prediction framework. The prediction set is derived from the entropy of the most frequent answer, ensuring theoretically guaranteed coverage. Results: Evaluated across six mainstream LLMs and four MCQA benchmarks, our approach significantly outperforms logit-based baselines, achieving substantial AUROC improvements while empirically satisfying the prescribed risk level for miscoverage. This provides a verifiable, theoretically sound solution for trustworthy reasoning in high-stakes applications.
📝 Abstract
Large Language Models (LLMs) have shown remarkable progress in multiple-choice question answering (MCQA), but their inherent unreliability, such as hallucination and overconfidence, limits their application in high-risk domains. To address this, we propose a frequency-based uncertainty quantification method under black-box settings, leveraging conformal prediction (CP) to ensure provable coverage guarantees. Our approach involves multiple independent samplings of the model's output distribution for each input, with the most frequent sample serving as a reference to calculate predictive entropy (PE). Experimental evaluations across six LLMs and four datasets (MedMCQA, MedQA, MMLU, MMLU-Pro) demonstrate that frequency-based PE outperforms logit-based PE in distinguishing between correct and incorrect predictions, as measured by AUROC. Furthermore, the method effectively controls the empirical miscoverage rate under user-specified risk levels, validating that sampling frequency can serve as a viable substitute for logit-based probabilities in black-box scenarios. This work provides a distribution-free model-agnostic framework for reliable uncertainty quantification in MCQA with guaranteed coverage, enhancing the trustworthiness of LLMs in practical applications.