Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

2026-03-09

AI Summary

This work addresses the issue of overconfidence in large language models trained via reinforcement learning with verifiable rewards (RLVR), where models often exhibit degraded calibration and assign high confidence to incorrect answers. The study is the first to reveal a gradient conflict between optimizing policy accuracy and minimizing calibration error. To resolve this, the authors propose the Decoupled Confidence and Policy Optimization (DCPO) framework, which disentangles confidence estimation from reasoning objectives and jointly optimizes both goals. Experiments demonstrate that DCPO achieves calibration performance significantly superior to existing methods while maintaining reasoning accuracy comparable to GRPO, thereby effectively mitigating overconfidence without compromising task performance.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) significantly enhances large language models (LLMs) reasoning but severely suffers from calibration degeneration, where models become excessively over-confident in incorrect answers. Previous studies devote to directly incorporating calibration objective into existing optimization target. However, our theoretical analysis demonstrates that there exists a fundamental gradient conflict between the optimization for maximizing policy accuracy and minimizing calibration error. Building on this insight, we propose DCPO, a simple yet effective framework that systematically decouples reasoning and calibration objectives. Extensive experiments demonstrate that our DCPO not only preserves accuracy on par with GRPO but also achieves the best calibration performance and substantially mitigates the over-confidence issue. Our study provides valuable insights and practical solution for more reliable LLM deployment.

Problem

• calibration degeneration
• over-confidence
• Reinforcement Learning from Verifiable Rewards
• large language models

Innovation

• Decoupling
• Calibration
• Reinforcement Learning from Verifiable Rewards
• Gradient Conflict
• Over-confidence Mitigation