Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards

2026-06-21

AI Summary

This work addresses the structural reward collapse induced by length penalties on incorrect answers in large reasoning models, which undermines reasoning capabilities. To mitigate this issue, the authors propose ACOER, a method that operates within the Group Relative Policy Optimization (GRPO) framework and introduces an adaptive conciseness reward exclusively for correct answers. ACOER further incorporates dynamic budget normalization and a feedback-controlled penalty adjustment mechanism to effectively prevent both structural and stochastic collapse. Experimental results demonstrate that ACOER consistently improves accuracy across multiple mathematical reasoning benchmarks while reducing generated tokens by over 60%, achieving efficient and stable reasoning optimization.

Abstract

Training large language models to reason efficiently is a critical challenge. While integrating length-penalizing rewards into Group Relative Policy Optimization (GRPO) aims to reduce verbosity, it frequently triggers reward collapse, severely degrading reasoning capabilities. Through a systematic evaluation of various reward configurations, we identify the root mechanism: GRPO's group normalization creates divergent advantages when incorrect answers receive continuous length penalties. Consequently, methods penalizing the length of incorrect answers are structurally prone to collapse under sustained optimization. Furthermore, restricting penalties exclusively to correct answers avoids this primary failure, but leaves the model susceptible to a stochastic collapse driven by response over-compression. To robustly prevent both failure modes, we propose ACOER (Adaptive Correct-Only Efficiency Reward). ACOER eliminates the structural penalty loop by isolating brevity bonuses to correct completions and prevents stochastic compression via dynamic budget normalization and control-loop penalty adjustments. Evaluated across diverse mathematical reasoning benchmarks, ACOER improves overall accuracy compared to the base model while reducing token generation by over 60%, establishing a fundamentally stable approach for efficiency-aware optimization.

Problem

• reward collapse
• reasoning efficiency
• length penalty
• large language models
• training stability

Innovation

• Adaptive Correct-Only Efficiency Reward
• reward collapse prevention
• efficiency-aware optimization
• length penalty
• Group Relative Policy Optimization