What Can You Do When You Have Zero Rewards During RL?

2025-10-04

AI Summary

In reinforcement learning (RL) for large language models (LLMs), persistent failure on complex reasoning tasks yields uniformly zero rewards, resulting in vanishing gradients and complete training stagnation—termed the “zero-reward barrier.” Method: To address this without modifying RL algorithms, we propose a data-centric intervention: injecting synthetically constructed easy examples—simplified task instances that the model can reliably solve—into the training set to bootstrap initial learning signals. Our approach operates within a graph-search task framework and integrates dense reward shaping, diversity-promoting incentives, and credit assignment mechanisms. Contribution/Results: Experiments demonstrate that standard RL methods fail entirely under zero-reward conditions, whereas incorporating easy examples enables progressive improvement and eventual success on the original hard tasks. This work provides the first systematic empirical validation of easy examples as an effective mechanism for overcoming zero-reward bottlenecks in LLM reasoning. We open-source multiple baseline implementations, establishing a new paradigm for optimizing LLM reasoning under low- or sparse-reward regimes.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Reinforcement learning (RL) with outcome-based rewards has proven effective for improving large language models (LLMs) on complex reasoning tasks. However, its success often depends on the base model occasionally sampling correct solutions. When no correct solutions are sampled, training encounters a zero-reward barrier where learning stalls due to zero gradients. We study this scenario through the graph search task introduced in Bachmann et al. (2024) and evaluate recent methods that incorporate desirable components such as dense rewards, diversity incentives, and improved credit assignment. Our experiments show that none of these approaches overcome the zero-reward barrier if the base model never produces a correct answer. In contrast, we find that a simple data-centric intervention of adding easier samples to the training set enables the model to eventually solve the original hard task despite starting from zero reward. Importantly, this succeeds without modifying the RL algorithm itself. Because official implementations of several baselines were unavailable, we developed our own, which allowed us to conduct a detailed analysis of their failure modes. We release these implementations to support further research at: https://github.com/rl4reasoning/rl-baselines

Problem

• Addresses zero-reward barrier in RL training
• Overcomes learning stall when no correct solutions sampled
• Enables solving hard tasks via data-centric intervention

Innovation

• Adding easier samples to training set
• Overcoming zero-reward barrier without RL modification
• Enabling solution of hard tasks from zero reward