Leveraging Error Diversity in Group Rollouts for Reinforcement Learning

2026-05-17

AI Summary

Existing reinforcement learning approaches often overlook the diversity of error distributions in population-based sampling, leading to error entrenchment and suboptimal training efficiency. This work proposes a lightweight, algorithm-agnostic advantage signal modulation mechanism that, for the first time, leverages intra-population error diversity as a training signal to dynamically adjust the penalty strength applied to erroneous trajectories, thereby encouraging diverse reasoning paths. Built upon the RLVR framework, the method employs a post-hoc advantage shaping strategy, enabling plug-and-play compatibility with mainstream reinforcement learning algorithms. Evaluated across seven mathematical reasoning benchmarks, it consistently improves performance across various models and RLVR variants, achieving an average gain of 6.29 points over DAPO on Qwen3-8B.

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) typically samples multiple responses per prompt and assigns binary rewards based on individual correctness, yet the collective structure of the group output, specifically the distribution of errors, is largely discarded. We identify this as a missed opportunity: empirical analysis reveals that error diversity within a group is a strong predictor of training success, with problems eliciting diverse wrong answers benefiting substantially more from RLVR than those producing homogeneous failures. Motivated by this observation, we propose Error Diversity Advantage Shaping (EDAS), a lightweight, algorithm-agnostic technique that modulates the advantage signal for incorrect rollouts based on intra-group error diversity. EDAS amplifies penalties for dominant, repeated errors and attenuates penalties for rare, exploratory ones, thereby encouraging the model to maintain diverse reasoning paths and discouraging error perseveration. Crucially, EDAS operates as a simple post-hoc adjustment that can be seamlessly integrated into any RLVR algorithm. We validate EDAS on top of several mainstream RLVR methods across a series of models and seven challenging math benchmarks, demonstrating consistent improvements. Notably, EDAS yields an average improvement of 6.29 points over DAPO on Qwen3-8B across seven benchmarks, confirming that exploiting the latent information in group rollouts is a broadly effective strategy for strengthening RLVR.

Problem

• Reinforcement Learning from Verifiable Rewards
• error diversity
• group rollouts
• advantage shaping
• reward signal

Innovation

• Error Diversity
• Reinforcement Learning from Verifiable Rewards
• Advantage Shaping
• Group Rollouts
• Algorithm-Agnostic