MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning

2026-02-19

AI Summary

This work addresses the limitations of existing reinforcement learning with value-based rewards (RLVR) algorithms—such as GRPO—which rely on rigid, uniform, and symmetric trust-region mechanisms that poorly adapt to the complex optimization dynamics of large language models. These constraints lead to inefficient gradient utilization, insensitivity to probability mass, and asymmetric signal reliability. To overcome these issues, we propose the MASPO framework, which unifies optimization across three critical dimensions: gradient efficiency, probability mass awareness, and signal reliability. Specifically, MASPO employs a differentiable soft Gaussian gating mechanism to enhance gradient efficiency, introduces a quality-adaptive clipping strategy to balance exploration, and designs an asymmetric risk controller to align update magnitudes with signal confidence. Experiments demonstrate that MASPO significantly outperforms strong baselines, achieving superior performance, robustness, and sample efficiency across multiple benchmarks.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Existing Reinforcement Learning with Verifiable Rewards (RLVR) algorithms, such as GRPO, rely on rigid, uniform, and symmetric trust region mechanisms that are fundamentally misaligned with the complex optimization dynamics of Large Language Models (LLMs). In this paper, we identify three critical challenges in these methods: (1) inefficient gradient utilization caused by the binary cutoff of hard clipping, (2) insensitive probability mass arising from uniform ratio constraints that ignore the token distribution, and (3) asymmetric signal reliability stemming from the disparate credit assignment ambiguity between positive and negative samples. To bridge these gaps, we propose Mass-Adaptive Soft Policy Optimization (MASPO), a unified framework designed to harmonize these three dimensions. MASPO integrates a differentiable soft Gaussian gating to maximize gradient utility, a mass-adaptive limiter to balance exploration across the probability spectrum, and an asymmetric risk controller to align update magnitudes with signal confidence. Extensive evaluations demonstrate that MASPO serves as a robust, all-in-one RLVR solution, significantly outperforming strong baselines. Our code is available at: https://anonymous.4open.science/r/ma1/README.md.

Problem

• gradient utilization
• probability mass
• signal reliability
• trust region
• credit assignment

Innovation

• Soft Policy Optimization
• Gradient Utilization
• Probability Mass Adaptation
• Asymmetric Signal Reliability
• RLVR