LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models

2026-05-19

AI Summary

This work addresses the limitations of traditional group-based relative policy optimization methods—such as GRPO—that rely on a single statistical baseline (e.g., group mean) and thus fail to capture fine-grained trajectory-level preferences, particularly in reward settings sensitive to ranking. To overcome this, the authors propose LambdaPO, a novel framework that reformulates advantage estimation as a pairwise preference structure. LambdaPO jointly optimizes policy updates through dynamic confidence-weighted inter-trajectory reward differences and a semantic density reward, eliminating the need for an explicit value critic. The method innovatively decomposes the advantage function into confidence-modulated integrals over pairwise preferences and introduces a precision-recall-aligned semantic density reward to preserve richer optimization signals. Experiments demonstrate that LambdaPO significantly outperforms existing baselines across multiple mathematical reasoning and question-answering tasks, effectively enhancing the complex reasoning capabilities of large language models.

Abstract

Group Relative Policy Optimization(GRPO) has become a cornerstone of modern reinforcement learning alignment, prized for its efficacy in foregoing an explicit value-critic by leveraging reward normalization across sampled trajectory cohorts. However, the method's reliance on a monolithic statistical baseline, such as the group mean, collapses the relational topology of the trajectory space into a single scalar, thereby erasing the fine-grained preference information essential for navigating complex, rank-sensitive reward landscapes. To address this issue, we introduce a novel framework, Lambda Policy Optimization (LambdaPO), that addresses this information-theoretic bottleneck by re-conceptualizing advantage estimation from a scalar value to a decomposed, pairwise preference structure. Specifically, the advantage for any given trajectory is formulated as the integrated sum of reward differentials against all peers in its cohort, where each pairwise comparison is dynamically attenuated by the policy's own probabilistic confidence in the established preference. To further mitigate the sparsity of binary outcome supervision, we augment the objective with a semantic density reward, derived from the precision-recall alignment between generated reasoning traces and ground-truth solutions. As a result, our method can mine more fine-grained optimization signals from a group of rollouts, guiding the LLM to a better optima. Experimental results across challenging math reasoning and question-answering tasks demonstrates that LambdaPO improves performance compared to the baseline methods.

Problem

• relative policy optimization
• reward normalization
• preference information
• trajectory space
• fine-grained optimization

Innovation

• Lambda Policy Optimization
• pairwise preference
• advantage estimation
• semantic density reward
• reasoning language models