Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration

2026-04-13

AI Summary

This work addresses the high computational cost and slow convergence of existing reinforcement learning with human feedback (RLHF) or reward-based learning (RLVR) methods for training large language models, as well as the inadequacy of linear extrapolation in accurately modeling parameter evolution. The authors observe that LoRA parameter trajectories during RLVR training exhibit pronounced nonlinear dynamics within a rank-1 subspace. Building on this insight, they propose NExt, a novel framework that introduces, for the first time, a nonlinear extrapolation mechanism for low-rank parameter trajectories. NExt constructs a nonlinear predictor by extracting the dominant rank-1 subspace to efficiently model and extrapolate parameter updates. Evaluated across multiple RLVR algorithms and tasks, the method reduces computational overhead by 37.5% on average while preserving training performance and compatibility.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recently, scaling reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs) has emerged as an effective training paradigm for significantly improving model capabilities, which requires guiding the model to perform extensive exploration and learning, leading to substantial computational overhead and becoming a key challenge. To reduce the number of training steps, Prior work performs linear extrapolation of model parameters. However, the dynamics of model parameter updates during RLVR training remain insufficiently understood. To further investigate the evolution of LLMs during RLVR training, we conduct empirical experiments and find that the rank-1 subspace of the model does not evolve linearly, and its dominance over the original parameters is further amplified during LoRA training. Based on the above insights, we propose the \textbf{N}onlinear \textbf{Ext}rapolation of low-rank trajectories (\textbf{NExt}), a novel framework that models and extrapolates low-rank parameter trajectories in a nonlinear manner. Concretely, we first train the model using LoRA and extract the rank-1 subspace of parameter differences at multiple training steps, which is then used for the subsequent nonlinear extrapolation. Afterward, we utilized the extracted rank-1 subspace to train a predictor, which can model the trajectory of parameter updates during RLVR, and then perform the predict-extend process to extrapolate model parameters, achieving the acceleration of RLVR. To further study and understand NExt, we conduct comprehensive experiments that demonstrate the effectiveness and robustness of the method. Our method reduces computational overhead by approximately 37.5\% while remaining compatible with a wide range of RLVR algorithms and tasks. We release our code in https://github.com/RUCAIBox/NExt.

Problem

• RLVR
• Large Language Models
• Low-rank Optimization
• Training Acceleration
• Computational Overhead

Innovation

• low-rank optimization
• nonlinear extrapolation
• RLVR acceleration
• LoRA
• parameter trajectory modeling