Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation

2026-05-01

AI Summary

This study systematically evaluates the effectiveness of test-case pass-rate rewards—a commonly adopted dense reward signal—in critic-free reinforcement learning for code generation. Despite its promise in mitigating reward sparsity, empirical analysis across multiple base models and algorithms (including GRPO and RLOO) reveals limited performance gains over simple binary rewards. Through controlled experiments and gradient direction analysis, the work demonstrates that pass-rate rewards, while denser, suffer from miscalibration and intra-batch gradient conflicts that impede convergence to fully correct solutions. Consequently, the observed improvements are not statistically significant compared to binary success/failure signals, casting doubt on the practical utility of pass-rate rewards within current reinforcement learning frameworks for code generation.

Abstract

Reinforcement learning (RL) from unit-test feedback has become a standard post-training recipe for improving large language models (LLMs) on code generation. However, the pass-all-tests binary reward can be sparse, yielding no learning signal on challenging problems where none of the sampled solutions passes all tests. A common remedy is to use the test-case pass rate as a surrogate reward. In this work, we study pass-rate rewards in critic-free RL for code generation (e.g., GRPO and RLOO) and report a consistent pattern across base models and algorithms: despite alleviating reward sparsity, pass-rate rewards do not reliably improve final performance over binary rewards in rigorous controlled experiments. To understand this discrepancy, we analyze reward density and the resulting gradient directions. We find that pass-rate rewards are denser, but the induced gradient updates do not consistently move probability mass toward full-pass solutions. This arises because test-case pass rate is a miscalibrated surrogate for progress toward full correctness, and partial-pass solutions within the same group can induce conflicting gradient directions that cancel out. Overall, our results suggest that, in critic-free RL, pass-rate rewards are insufficient to improve code generation and motivate reward designs that better align optimization with the goal of full correctness.

Problem

• pass-rate reward
• reinforcement learning
• code generation
• reward sparsity
• full correctness

Innovation

• pass-rate reward
• reinforcement learning
• code generation
• reward sparsity
• gradient alignment