Rethinking the Comparison Unit in Sequence-Level Reinforcement Learning: An Equal-Length Paired Training Framework from Loss Correction to Sample Construction

2026-04-19

AI Summary

This work addresses a critical optimization bias in sequence-level reinforcement learning arising from the inherent incomparability of response pairs with mismatched lengths. The authors attribute this issue to non-comparable comparison units at the sample construction level and propose EqLen, an equal-length paired training framework. EqLen actively constructs aligned, comparable training segments of uniform length during generation through dual-track synchronous decoding, prefix inheritance, and segment masking. This approach is compatible with existing algorithms such as GRPO, GSPO, and RLOO, effectively mitigating length-induced bias and enabling more stable and efficient sequence-level reinforcement learning.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Reinforcement Learning
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper investigates the length problem in sequence-level relative reinforcement learning. We observe that, although existing methods partially alleviate length-related phenomena, a more fundamental issue remains insufficiently characterized: the comparison units used during training lack inherent comparability. Building on this observation, we propose a new perspective: the length problem should not be viewed merely as a loss-scaling or normalization bias, but rather as a \emph{comparison unit construction} problem. We further establish a sample-construction-based training framework that, instead of applying post-hoc corrections to unequal-length responses, proactively constructs equal-length, alignable, and comparable training segments during generation. Within this framework, we propose EqLen, a concrete method applicable to group-relative comparison algorithms such as GRPO, GSPO, and RLOO. Through dual-track synchronous generation, prefix inheritance, and segment masking, EqLen efficiently collects effective equal-length training segments and enables stable

Problem

• sequence-level reinforcement learning
• comparison unit
• length problem
• relative comparison
• training framework

Innovation

• comparison unit construction
• equal-length training
• sequence-level reinforcement learning
• sample construction
• EqLen