On Predictability of Reinforcement Learning Dynamics for Large Language Models

2025-10-01

AI Summary

The parameter dynamics of large language models (LLMs) during reinforcement learning (RL) training remain poorly understood. Method: This work first reveals that parameter updates in RL fine-tuning exhibit strong rank-one dominance and linear evolution, with the low-rank subspace—extracted via singular value decomposition (SVD)—accurately captured by early checkpoints. Leveraging this insight, we propose AlphaRL, a parameter-efficient acceleration framework that requires no auxiliary modules or hyperparameter tuning, and extrapolates parameter evolution via linear dynamical modeling. Contribution/Results: Evaluated across eight mainstream LLMs and seven RL algorithms, AlphaRL achieves an average 2.0× speedup (up to 2.5×) while preserving ≥96% of the original inference performance. The framework demonstrates broad applicability and practical utility without compromising model quality.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in reasoning capabilities of large language models (LLMs) are largely driven by reinforcement learning (RL), yet the underlying parameter dynamics during RL training remain poorly understood. This work identifies two fundamental properties of RL-induced parameter updates in LLMs: (1) Rank-1 Dominance, where the top singular subspace of the parameter update matrix nearly fully determines reasoning improvements, recovering over 99% of performance gains; and (2) Rank-1 Linear Dynamics, where this dominant subspace evolves linearly throughout training, enabling accurate prediction from early checkpoints. Extensive experiments across 8 LLMs and 7 algorithms validate the generalizability of these properties. More importantly, based on these findings, we propose AlphaRL, a plug-in acceleration framework that extrapolates the final parameter update using a short early training window, achieving up to 2.5 speedup while retaining extgreater 96% of reasoning performance without extra modules or hyperparameter tuning. This positions our finding as a versatile and practical tool for large-scale RL, opening a path toward principled, interpretable, and efficient training paradigm for LLMs.

Problem

• Understanding RL parameter dynamics in large language models
• Identifying rank-1 dominance in RL-induced parameter updates
• Predicting training outcomes from early checkpoints for acceleration

Innovation

• Identifies rank-1 dominance in RL updates
• Proposes linear dynamics for parameter prediction
• Develops AlphaRL framework for accelerated training