Confidence-Aware Alignment Makes Reasoning LLMs More Reliable

2026-05-08

AI Summary

Large language models often arrive at correct answers through flawed reasoning paths, leading to a disconnect between accuracy and reasoning reliability. To address this, this work proposes the CASPO framework, which aligns token-level confidence with stepwise logical correctness via iterative direct preference optimization and introduces Confidence-aware Thinking (CaT) to dynamically prune low-confidence reasoning branches during inference. Notably, CASPO achieves joint confidence calibration and dynamic reasoning pruning without requiring an additional reward model. Experimental results demonstrate that CASPO significantly enhances both reasoning reliability and efficiency across ten benchmarks and multiple model families. In particular, Qwen3-8B-Base outperforms tree-search baselines on AIME'24 and AIME'25 without relying on reward-model-derived data.

Abstract

Large reasoning models often reach correct answers through flawed intermediate steps, creating a gap between final accuracy and reasoning reliability. Existing alignment strategies address this with external verifiers or massive sampling, limiting scalability. In this work, we introduce CASPO (Confidence-Aware Step-wise Preference Optimization), a framework that aligns token-level confidence with step-wise logical correctness through iterative Direct Preference Optimization, without training a separate reward model. During inference, we propose Confidence-aware Thought (CaT), which leverages this calibrated confidence to dynamically prune uncertain reasoning branches with negligible O(V) latency. Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency. CASPO scales to Qwen3-8B-Base and surpasses tree-search baselines on AIME'24 and AIME'25 without using reward-model data. We also release a step-wise dataset with confidence annotations to support fine-grained analysis of reasoning reliability. Code is available at https://github.com/Thecommonirin/CASPO.

Problem

• reasoning reliability
• confidence calibration
• step-wise correctness
• large language models
• alignment

Innovation

• Confidence-Aware Alignment
• Step-wise Preference Optimization
• Direct Preference Optimization
• Reasoning Reliability
• Dynamic Pruning