Rethinking Groups in Critic-Free RLVR

2026-06-15

AI Summary

This work addresses the limitations of existing critic-free reinforcement learning methods, which rely on multi-pass grouping for training, resulting in low data efficiency, synchronization challenges, and poor compatibility with structured outputs. The authors re-examine the fundamental role of grouping and argue that its primary purpose is to prevent incorrect penalization of negative samples. Building on this insight, they propose a negative token filtering strategy that enables effective single-pass training without explicit grouping. Combined with batched advantage estimation, the proposed approach significantly enhances training flexibility and efficiency. Empirical results demonstrate that it matches the performance of grouped RL on reasoning tasks and achieves superior results on agent-based tasks.

Abstract

Reinforcement learning (RL) has become a central paradigm for post-training large language models. Existing critic-free RL methods typically generate a group of rollouts for the same question to estimate value baselines for advantage computation. However, this design suffers from data inefficiency, group synchronization barriers, and inflexibility with structured rollouts. In this work, we revisit the role of the ``group'' and show that its underlying function is not merely to estimate baselines but to prevent false penalties on negative samples. Building on this insight, we propose negative token filtering, a simple and effective strategy that enables stable single-rollout training. We apply it to two batch-level advantage methods, achieving comparable performance on reasoning tasks and stronger performance on agentic tasks relative to group-based RL techniques.

Problem

• critic-free RL
• data inefficiency
• group synchronization
• structured rollouts
• advantage estimation

Innovation

• critic-free RL
• negative token filtering
• single-rollout training
• advantage estimation
• group sampling