CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoning

2026-05-22

AI Summary

This work addresses the lack of a systematic understanding of optimal weight assignment principles in prompt reweighting methods for large language model reinforcement learning. It proposes CurveRL, which formulates reweighting as the functional derivative of a utility functional over the space of pass-rate functions, establishing a unified optimality framework. The method introduces a novel distribution-aware reweighting mechanism based on quantile coordinate transformation, weighting prompts according to their rank and density in the pass-rate distribution rather than absolute values. This reveals that contextual distribution control is the core principle underlying effective reweighting design. Experimental results demonstrate that CurveRL significantly outperforms existing RLVR approaches such as GRPO across multiple benchmarks, validating the efficacy and superiority of the proposed mechanism.

Abstract

Context or prompt-level reweighting has emerged as a central algorithmic lever in Reinforcement Learning with Verified Rewards (RLVR) for improving the reasoning capability of large language models, yet the principle determining what constitutes an optimal weighting remains poorly understood. We address this gap by formulating prompt reweighting as a functional derivative of a utility functional defined in the pass-rate function space, yielding a unified optimality framework that accommodates existing schemes, including REINFORCE and GRPO. Building on this optimality framework, we propose a distribution-aware prompt reweighting approach, called CurveRL, based on a quantile coordinate transform, in which the weight assigned to each prompt depends not on the absolute value of pass rates but on its rank and density to reflect the distributional structure of the pass rates in the learning dynamics. Extensive experiments across multiple benchmarks demonstrate that our proposed CurveRL consistently outperforms GRPO and other RLVR baselines. Our study identifies context-distribution control as a principled axis for analyzing and designing prompt-reweighted RLVR algorithms. The code is released in https://github.com/zhyzmath/CurveRL.

Problem

• prompt reweighting
• reinforcement learning
• large language models
• reasoning
• distribution-aware

Innovation

• distribution-aware reweighting
• functional derivative
• quantile coordinate transform
• RLVR
• prompt reweighting