When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study

2026-06-18

AI Summary

This study addresses the challenge of high-cost ground-truth reward signals in code generation by systematically evaluating confidence-based intrinsic reward strategies. Through large-scale experiments on LiveCodeBench, the work reveals a previously undocumented phenomenon: such methods consistently yield early performance gains but inevitably collapse during later training stages. This collapse is shown to be highly sensitive to sample size and temperature parameters. The authors compare deterministic Reinforcement Learning with Intrinsic Feedback (RLIF), majority voting, confidence scoring, and their combinations with verifiable reward-based Reinforcement Learning (RLVR). Results indicate that RLIF pretraining fails to substantially enhance RLVR performance and that models driven solely by intrinsic rewards progressively shorten their outputs and lose reasoning capabilities.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in large language model reasoning, but relies on ground-truth supervision that is costly or infeasible, especially in coding tasks. Recent work addresses this by deriving rewards from a model's own signals, such as majority voting or confidence-based scores, achieving notable success on mathematical reasoning benchmarks. However, code generation poses distinct challenges: programs are structurally complex, semantically equivalent solutions may differ syntactically, and verification typically requires execution. Whether these intrinsic reward methods transfer effectively to code remains unexplored. In this work, we present a systematic empirical study of intrinsic reward methods for code generation. We conduct extensive experiments on LiveCodeBench, systematically evaluating representative certainty-based Reinforcement Learning from Internal Feedback (RLIF) approaches under different training scenarios and hyperparameter settings. Our experiments reveal that certainty-based methods yield early gains but inevitably collapse: models progressively shorten outputs and lose reasoning capability, with collapse speed sensitive to sample size and temperature. When used to initialize RLVR training, RLIF pre-training offers no significant improvement over training from scratch. We also provide actionable recommendations for using intrinsic rewards for training code reasoning models. Our study shows both the promise and limitations of intrinsic reward methods for code, informing future work on code models and agents.

Problem

• intrinsic rewards
• code generation
• reinforcement learning
• reasoning
• verification

Innovation

• intrinsic rewards
• code generation
• reinforcement learning
• output collapse
• RLIF