Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework

2026-05-21

AI Summary

This work addresses critical limitations in existing reinforcement learning from internal feedback (RLIF) methods, which rely on a single reward signal and are prone to reward hacking, entropy collapse, and degradation of reasoning structures. To overcome these issues, the authors propose a multi-reward RLIF framework that introduces, for the first time, a complementary dual internal reward mechanism: an answer-level reward derived from clustering-based voting and a completion-level reward based on per-token self-confidence. The approach further integrates GDPO normalization and a novel KL-Cov regularization to effectively mitigate entropy collapse in late-stage training. Evaluated on mathematical reasoning and code generation benchmarks, the method substantially outperforms current unsupervised RL approaches and achieves performance approaching that of externally supervised RLVR, significantly enhancing the stability and efficacy of unsupervised long-horizon reasoning.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of LLMs, but often depends on external supervision from human annotations or gold-standard solutions. Reinforcement learning from internal feedback (RLIF) has recently emerged as a scalable unsupervised alternative, using signals extracted from the model itself. However, existing RLIF methods typically rely on a single internal reward, which can lead to reward hacking, entropy collapse, and degraded reasoning structure. We propose a multi-reward RLIF framework that decomposes the training signal into two complementary components: an answer-level reward based on cluster voting and a completion-level reward based on token-wise self-certainty. To combine these signals robustly, we apply GDPO-based normalization to reduce reward-scale imbalance. We further introduce KL-Cov regularization, which targets low-entropy token distributions responsible for disproportionate entropy reduction, preserving exploration and preventing late-stage collapse. Across mathematical reasoning and code-generation benchmarks, our method improves stability and robustness over prior unsupervised RL approaches, while achieving performance close to supervised RLVR methods. These results show that complementary internal rewards, combined with targeted regularization, can support stable long-horizon reasoning without relying on external ground-truth supervision. Code will be released soon.

Problem

• reward hacking
• entropy collapse
• reinforcement learning from internal feedback
• reasoning degradation
• single-reward RLIF

Innovation

• multi-reward RLIF
• entropy collapse prevention
• internal feedback reinforcement learning
• KL-Cov regularization
• unsupervised reasoning