Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control

2025-08-07

AI Summary

Large language models (LLMs) frequently exhibit hallucination and produce unreliable outputs in multiple-choice question answering (MCQA). Method: This paper proposes the first trustworthy reasoning framework for MCQA that jointly integrates statistical significance testing and calibration-preserving prediction. It constructs a response distribution via self-consistent sampling, uses response frequency as a test statistic for p-value computation, and builds a minimum prediction set with theoretically guaranteed miscoverage rate ≤ α via empirical risk control. Contribution/Results: It is the first work to introduce hypothesis testing into LLM uncertainty quantification, ensuring strict calibration of prediction sets; it further proves that prediction set size serves as a valid uncertainty measure. Experiments on MMLU and MMLU-Pro demonstrate precise α-level miscoverage control and monotonic reduction of prediction set size with decreasing α, significantly enhancing reliability and interpretability—especially in high-risk scenarios.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This study introduces a significance testing-enhanced conformal prediction (CP) framework to improve trustworthiness of large language models (LLMs) in multiple-choice question answering (MCQA). While LLMs have been increasingly deployed in disciplinary QA scenarios, hallucination and nonfactual generation substantially compromise response reliability. Although CP provides statistically rigorous marginal coverage guarantees for prediction sets, and significance testing offers established statistical rigor, their synergistic integration remains unexplored. To mitigate hallucination and factual inaccuracies, our framework integrates $p$-value computation with conformity scoring through self-consistency resampling of MCQA responses. This approach calculates option frequencies to address LLMs' black-box nature, subsequently constructing prediction sets via null hypothesis testing ($mathcal{H}_0$) with empirically derived $p$-values. Evaluations on MMLU and MMLU-Pro benchmarks using off-the-shelf LLMs demonstrate: (1) The enhanced CP achieves user-specified empirical miscoverage rates; (2) Test-set average prediction set size (APSS) decreases monotonically with increasing risk levels ($α$), validating APSS as an effective uncertainty metric. This work establishes a principled statistical framework for trustworthy LLM deployment in high-stakes QA applications.

Problem

• Improving trustworthiness of LLMs in multiple-choice question answering
• Mitigating hallucination and factual inaccuracies in LLM responses
• Providing statistical guarantees for prediction sets in QA tasks

Innovation

• Conformal prediction with significance testing
• Self-consistency resampling for p-values
• Option frequency-based null hypothesis testing