Staleness-Learning Rate Scaling Laws for Asynchronous RLHF

2026-07-01

AI Summary

This work addresses the instability in asynchronous reinforcement learning from human feedback (RLHF), where stale rollouts introduce policy update bias and training divergence. For the first time, it explicitly models the behavior policy and, under distributional and policy smoothness assumptions, employs surrogate objective analysis, total derivative decomposition, and gradient bias bounding to establish a quantitative relationship among learning rate, maximum rollout delay, and training stability. Theoretically, it proves that the one-step gradient bias scales as O(S·η), where S denotes state-action sensitivity and η the learning rate, and derives a collapse-time scaling law that explains the empirically observed weak dependence of learning rate on delay under finite horizons. Building on these insights, the paper proposes a dual-constraint stability condition, offering practical and scalable guidance for stable asynchronous RLHF training.

Abstract

High-throughput RLHF systems often decouple rollout generation from policy optimization, leading to the use of stale rollouts during learner updates. In this work, we study the effect of such staleness in asynchronous GRPO. We make the behavior policy explicit in the GRPO surrogate objective and distinguish between the surrogate-gradient mapping used by the learner and the true total derivative of a distribution-dependent population objective. Under assumptions of local boundedness, distributional smoothness, and behavior-policy smoothness, we show that stale rollouts introduce a per-step surrogate-gradient bias of order O(S * eta), where S denotes the maximum rollout lag and eta denotes the learning rate. We further derive a conditional collapse-time scaling law: when within-cycle drift remains below a batch-level clipping radius, collapse is governed primarily by cumulative learner drift T * eta; when the stale-rollout constraint is active, stability instead depends explicitly on S * eta. This yields a two-constraint stability condition eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)}, explaining why the maximum stable learning rate may appear weakly dependent on staleness in the horizon-limited regime.

Problem

• staleness
• asynchronous RLHF
• learning rate scaling
• surrogate gradient bias
• training stability

Innovation

• staleness
• learning rate scaling
• asynchronous RLHF
• GRPO
• surrogate-gradient bias