ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models

2026-05-12

AI Summary

Large language models often exhibit high confidence even when generating incorrect answers, lacking reliable and interpretable uncertainty quantification. This work proposes a decoupled, sequence-aware verbal confidence calibration framework that first generates an answer and then independently estimates confidence based on the fixed question-answer pair, thereby avoiding interference with the original generation process. The method constructs a proxy signal for correctness likelihood through multiple sampling and introduces a ranking-aware reinforcement learning objective to align the relative ordering of verbal confidence with answer correctness. Experimental results demonstrate that the approach significantly improves calibration performance and error detection capability on both reasoning and knowledge-intensive tasks, while preserving the original answer accuracy with minimal degradation.

Abstract

Large language models (LLMs) often produce answers with high certainty even when they are incorrect, making reliable confidence estimation essential for deployment in real-world scenarios. Verbalized confidence, where models explicitly state their confidence in natural language, provides a flexible and user-facing uncertainty signal that can be applied even when token logits are unavailable. However, existing verbalized-confidence methods often optimize answer generation and confidence generation jointly, which can cause confidence-alignment objectives to interfere with answer accuracy. In this work, we propose a decoupled and order-aware framework for verbalized confidence calibration. Our method first generates an answer and then estimates confidence conditioned on the fixed question--answer pair, allowing confidence optimization without directly perturbing the answer-generation process. To align confidence with correctness likelihood, we construct a sampling-based surrogate from multiple model completions and optimize rank-based reinforcement learning objectives that encourage responses with higher estimated correctness likelihood to receive higher verbalized confidence. Experiments on reasoning and knowledge-intensive benchmarks show that our method improves calibration and failure prediction performance while largely preserving answer accuracy. These results demonstrate that verbalized confidence can be more reliably aligned by decoupling confidence estimation from answer generation and optimizing the relative ordering of confidence across responses.

Problem

• verbalized confidence
• confidence calibration
• large language models
• uncertainty estimation
• answer accuracy

Innovation

• verbalized confidence
• decoupled calibration
• order-aware alignment
• rank-based reinforcement learning
• confidence estimation