Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training

2026-05-26

AI Summary

This work addresses the inefficiency in group-based policy optimization where uniform rollout budget allocation often wastes computational resources on low-informativeness prompts. The authors propose Pilot-Commit, a novel framework that, for the first time, enables online dynamic evaluation of prompt informativeness and adaptively allocates rollouts to high-value prompts while skipping low-signal samples. By decoupling the pilot evaluation phase from the main training stage, Pilot-Commit remains compatible with existing algorithms such as GRPO. Empirical results across multiple mathematical reasoning benchmarks and language models ranging from 1.5B to 14B parameters demonstrate that the method achieves baseline-level accuracy at significantly lower sampling costs, improving rollout efficiency by 1.9× over GRPO and 4.0× over DAPO.

Abstract

Reinforcement learning (RL) is the dominant paradigm for post-training large language models. However, in the online, on-policy setting, rollout generation dominates the computational cost of training. Group-based policy optimization methods compute advantages from multiple rollouts per prompt, yet they indiscriminately allocate budget to prompts with collapsed reward distributions, wasting expensive rollouts on negligible learning signals. We demonstrate that group-based updates are most effective in regimes of high reward variance. Since the policy evolves throughout training, prompt informativeness must be estimated online rather than precomputed, but exhaustively evaluating every prompt is computationally prohibitive. We introduce Pilot-Commit, a budget-aware rollout allocation framework for group-based RL post-training. Pilot-Commit decouples prompt evaluation from exploitation: a pilot stage estimates per-prompt informativeness using a fraction of the budget, and the remaining rollouts are allocated to high-leverage prompts while low-signal prompts are skipped. Across multiple math reasoning benchmarks and model scales from 1.5B to 14B parameters, Pilot-Commit matches baseline accuracy with significantly lower sampling costs, reaching target accuracy up to $1.9\times$ faster than GRPO and $4.0\times$ faster than DAPO in cumulative rollouts.

Problem

• rollout allocation
• group-based RL
• post-training
• reward variance
• computational budget

Innovation

• rollout allocation
• group-based RL
• budget-aware optimization
• Pilot-Commit
• online informativeness estimation