Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective

2025-05-23

AI Summary

Reinforcement learning (RL) for enhancing large language models’ (LLMs) reasoning capabilities suffers from low sample efficiency during rollout, unstable problem difficulty estimation, and misalignment between estimated difficulty and model capability. To address these issues, we propose the Capability-Difficulty Alignment Sampling (CDAS) framework. First, we derive a robust difficulty estimate by aggregating historical rollout performance discrepancies. Second, we formulate a fixed-point system to quantitatively characterize the model’s current reasoning capability. Third, we implement dynamic weighted sampling to achieve real-time alignment between capability and difficulty, integrated with RLHF-style policy optimization. CDAS significantly improves training stability and convergence speed. Empirically, it achieves state-of-the-art average accuracy across multiple mathematical reasoning benchmarks and attains 2.33× higher training efficiency than the prior SOTA method, Dynamic Sampling.

Technology Category

• Machine Learning: Reinforcement Learning
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement learning exhibits potential in enhancing the reasoning abilities of large language models, yet it is hard to scale for the low sample efficiency during the rollout phase. Existing methods attempt to improve efficiency by scheduling problems based on problem difficulties. However, these approaches suffer from unstable and biased estimations of problem difficulty and fail to capture the alignment between model competence and problem difficulty in RL training, leading to suboptimal results. To tackle these limitations, this paper introduces extbf{C}ompetence- extbf{D}ifficulty extbf{A}lignment extbf{S}ampling ( extbf{CDAS}), which enables accurate and stable estimation of problem difficulties by aggregating historical performance discrepancies of problems. Then the model competence is quantified to adaptively select problems whose difficulty is in alignment with the model's current competence using a fixed-point system. Experimental results across a range of challenging mathematical benchmarks show that CDAS achieves great improvements in both accuracy and efficiency. CDAS attains the highest average accuracy against baselines and exhibits significant speed advantages compared to Dynamic Sampling, a competitive strategy in DAPO, which is extbf{2.33} times slower than CDAS.

Problem

• Improving sample efficiency in RL for LLM reasoning
• Aligning model competence with problem difficulty dynamically
• Accurate stable difficulty estimation via historical performance

Innovation

• Accurate difficulty estimation via historical performance discrepancies
• Quantifies model competence for adaptive problem selection
• Aligns problem difficulty with model competence dynamically