ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

2026-04-30

AI Summary

This work addresses a critical limitation in existing reinforcement learning–based methods for large language model (LLM) inference, wherein an overreliance on positive samples undermines generation diversity, while negative-sample training often suppresses shared semantic structures between positive and negative responses. To resolve this, the authors propose ResRL, a novel approach that constructs a low-rank subspace from positive samples via singular value decomposition (SVD), projects negative-sample hidden states onto this subspace, and modulates negative gradients using the projection residuals. This enables decoupling of shared semantics between positive and negative responses while preserving diversity. The method further introduces Lazy Likelihood Displacement as a single-forward proxy metric to enable conservative advantage reweighting. Evaluated across 12 benchmarks spanning mathematical reasoning, code generation, agent tasks, and function calling, ResRL significantly outperforms strong baselines, achieving a 9.4% gain in Avg@16 and a 7.0% improvement in Pass@128 on mathematical reasoning.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs) but usually exhibits limited generation diversity due to the over-incentivization of positive rewards. Although methods like Negative Sample Reinforcement (NSR) mitigate this issue by upweighting penalty from negative samples, they may suppress the semantic distributions shared between positive and negative responses. To boost reasoning ability without losing diversity, this paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses. We theoretically link Lazy Likelihood Displacement (LLD) to negative-positive head-gradient interference and derive a single-forward proxy that upper-bounds representation alignment to guide conservative advantage reweighting. ResRL then projects negative-token hidden representations onto an SVD-based low-rank positive subspace and uses projection residuals to modulate negative gradients, improving reasoning while preserving diversity and outperforming strong baselines on average across twelve benchmarks spanning Mathematics, Code, Agent Tasks, and Function Calling. Notably, ResRL surpasses NSR on mathematical reasoning by 9.4\% in Avg@16 and 7.0\% in Pass@128. Code is available at https://github.com/1229095296/ResRL.git.

Problem

• Reinforcement Learning
• Large Language Models
• Reasoning
• Generation Diversity
• Negative Samples

Innovation

• Residual Reinforcement Learning
• Negative Sample Projection
• Semantic Decoupling
• Lazy Likelihood Displacement
• SVD-based Subspace