Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR

2026-02-05

AI Summary

This work addresses a critical limitation in existing reinforcement learning with verifiable rewards (RLVR) methods—such as GSPO—where length bias during training leads to uncontrolled or even collapsed response lengths, thereby hindering reasoning performance. The study is the first to uncover the underlying mechanism driving this phenomenon and proposes Length-Unbiased Sequential Policy Optimization (LUSPO), a novel approach that stabilizes generation length by reformulating the loss function to eliminate length bias. Experimental results demonstrate that LUSPO significantly outperforms prior methods, including GRPO and GSPO, across both mathematical and multimodal reasoning benchmarks, establishing a new state-of-the-art optimization strategy in RLVR.

Technology Category

• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Sequential Decision Making
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent applications of Reinforcement Learning with Verifiable Rewards (RLVR) to Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated significant success in enhancing reasoning capabilities for complex tasks. During RLVR training, an increase in response length is often regarded as a key factor contributing to the growth of reasoning ability. However, the patterns of change in response length vary significantly across different RLVR algorithms during the training process. To provide a fundamental explanation for these variations, this paper conducts an in-depth analysis of the components of mainstream RLVR algorithms. We present a theoretical analysis of the factors influencing response length and validate our theory through extensive experimentation. Building upon these theoretical findings, we propose the Length-Unbiased Sequence Policy Optimization (LUSPO) algorithm. Specifically, we rectify the length bias inherent in Group Sequence Policy Optimization (GSPO), rendering its loss function unbiased with respect to response length and thereby resolving the issue of response length collapse. We conduct extensive experiments across mathematical reasoning benchmarks and multimodal reasoning scenarios, where LUSPO consistently achieves superior performance. Empirical results demonstrate that LUSPO represents a novel, state-of-the-art optimization strategy compared to existing methods such as GRPO and GSPO.

Problem

• response length variation
• length bias
• reinforcement learning with verifiable rewards
• reasoning capability
• sequence policy optimization

Innovation

• Length-Unbiased
• Sequence Policy Optimization
• Response Length Bias
• Reinforcement Learning with Verifiable Rewards
• LLM Reasoning