Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

2026-07-01

AI Summary

This work addresses the mismatch between confidence and accuracy in large language models trained with reinforcement learning, where reward signals based solely on answer correctness lead to overconfident hallucinations. To mitigate this issue, the authors propose C3RL, a novel algorithm that, for the first time, integrates confidence calibration directly into the reinforcement learning reward framework. C3RL jointly optimizes answer correctness, confidence calibration, and reference accuracy, and introduces a Calibration-aware Adaptive Strategy (CAS) that allocates reasoning resources adaptively based on calibrated confidence. Experiments across eight datasets demonstrate that C3RL significantly improves calibration performance without compromising accuracy. Furthermore, CAS consistently outperforms majority voting on both in-domain and out-of-domain tasks, reducing inference cost by up to 12.33×.

Abstract

Training large language models (LLMs) with reinforcement learning (RL) has significantly advanced their performance on reasoning and question-answering tasks. However, prevailing RL reward designs typically prioritize response correctness, neglecting to incentivize models to express their confidence accurately. This leads to a critical problem: performance gains are often accompanied by poor calibration between confidence and accuracy, misleading models to overconfidently hallucinate when uncertain. To address this limitation, we propose $\textbf{C}$orrectness and $\textbf{C}$onfidence $\textbf{C}$alibration $\textbf{R}$einforcement $\textbf{L}$earning ($\textbf{C3RL}$), a novel RL algorithm integrating correctness, calibration and dataset-informed reference accuracy rewards together. Comprehensive evaluation across 8 text and multimodal datasets demonstrates that C3RL enhances calibration without sacrificing accuracy, outperforming the current state-of-the-art method in both performance and calibration metrics. Utilizing the well-calibrated verbalized confidence from C3RL, we further introduce $\textbf{C}$onfidence-based $\textbf{A}$daptive Test Time $\textbf{S}$caling ($\textbf{CAS}$), an adjustable inference-time strategy that allocates computational resources based on response confidence. Experiments show that CAS surpasses majority voting on both in-domain and out-of-domain datasets while reducing the inference budget by up to 12.33 times. We believe the synergy of C3RL and CAS paves the way for deploying more reliable and resource-efficient LLMs. The code, data and models will be released.

Problem

• confidence calibration
• large language models
• reinforcement learning
• hallucination
• test-time scaling

Innovation

• confidence calibration
• reinforcement learning
• adaptive inference
• large language models
• test-time scaling