Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought

2026-04-20

AI Summary

This work addresses the gradient bias introduced by naive weighting of successful and failed attempts in existing reinforcement learning approaches for multi-attempt chain-of-thought reasoning. To mitigate this issue, the authors propose Calibrated Attempt-Level GRPO (CAL-GRPO), a method that employs an unbiased, low-variance attempt-weighting strategy combined with hard verification feedback and attempt-level reward modeling to optimize the Verification@K objective. Theoretical analysis elucidates how per-attempt rewards influence training dynamics and final performance. Experimental results demonstrate that CAL-GRPO significantly outperforms both the original GRPO and naive weighted baselines on both synthetic and real-world datasets, achieving state-of-the-art performance on the Verification@K metric and thereby validating both the efficacy of the proposed method and the underlying theoretical insights.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

State-of-the-art reasoning models utilize long chain-of-thought (CoT) to solve increasingly complex problems using more test-time computation. In this work, we explore a long CoT setting where the model makes up to K successive attempts at solving a problem, in which each attempt is allowed to build on earlier ones after the model receives a hard verifier feedback. This motivates RL methods that can harness per-attempt rewards by carefully weighting individual attempts. We study optimizing the Verification@K reward (the model succeeds by the K-th attempt) and show that naively weighing the attempts by their pass/fail results in biased gradients. We introduce Calibrated Attempt-Level (CAL) GRPO by devising a weighing strategy to obtain unbiased gradients while maintaining small variance. Our theory reveals how incorporating per-attempt rewards influence the training and the eventual Verification@K performance. Experiments, baselines, and ablations on synthetic and real data corroborate our theory and the benefits of CAL-GRPO over vanilla GRPO as well as naive weighting.

Problem

• Chain-of-Thought
• Reinforcement Learning
• Verification@K
• Gradient Bias
• Multi-Attempt Reasoning

Innovation

• Calibrated Reinforcement Learning
• Chain-of-Thought
• Verification@K
• Unbiased Gradient Estimation
• Multi-Attempt Reasoning