Predictable GRPO: A Closed-Form Model of Training Dynamics

2026-06-29

AI Summary

This work addresses the lack of theoretical understanding of GRPO training dynamics, which currently rely on empirical hyperparameter tuning. By leveraging first-principles reasoning, the authors formulate GRPO dynamics as a physical potential system influenced by an inertia term. Through dimensionality reduction, mean-field approximation, and softmax-bandit simplification, they derive a closed-form trajectory model that elucidates both the overdamped limit and oscillatory transition mechanisms. This model serves as a diagnostic tool capable of distinguishing among multiple failure modes. Empirical validation across three models and two group sizes demonstrates reward trajectory fits with R² ≥ 0.91, confirming group-size invariance and the predictability of stability thresholds.

Abstract

Group Relative Policy Optimization (GRPO) has become a standard tool for improving the reasoning ability of large language models, yet its training dynamics are still described empirically: reward trajectories are fit with low-parameter functional forms whose constants carry no mechanistic meaning, and hyperparameter choices remain a matter of trial and error. We develop a first-principles reduced-order model of these dynamics. The reduction has three consequences. First, it subsumes the empirical single-exponential saturation law as its overdamped limit, recasting the fitted plateau, timescale, and size exponent as the fixed point, inverse stiffness, and curvature-scaling exponent of the underlying potential, and adding, through the retained inertial term, the slow-start phase the single exponential cannot represent. Second, it yields predictions tied to independently measurable quantities rather than fitted ones: group-size invariance of the deterministic trajectory with a $1/G$ stationary fluctuation, a sharp stability threshold in the refresh interval, and an overdamped-to-oscillatory transition. Third, it furnishes diagnostics that separate failure modes a reward curve alone conflates -- reward hacking, advantage degeneracy, policy concentration, and dynamical instability. Across three models and two group sizes, the closed-form trajectory fits training reward to $R^2 \geq 0.91$ and the predicted group-size invariance holds on both the reward curve and out-of-distribution transfer to eight math benchmarks. The stability and oscillatory predictions are exercised in a controlled exact-reduction setting where the mean-field assumption holds exactly: a softmax-bandit reduction reproduces the predicted overdamped-to-oscillatory transition and locates the refresh-interval stability threshold at the independently measured stiffness, with a deep-network demonstration left to future work.

Problem

• GRPO
• training dynamics
• reward trajectory
• hyperparameter
• failure modes

Innovation

• GRPO
• closed-form model
• training dynamics
• stability threshold
• reward hacking