D$^2$Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning

2026-05-16

AI Summary

This work addresses the challenges in applying reinforcement learning to large language model (LLM) reasoning, where effective training data is scarce and problem difficulty dynamically shifts—particularly due to the paucity of medium-difficulty samples that rapidly become obsolete as model capabilities evolve. To overcome this, we propose a dual difficulty-aware self-evolution framework that introduces, for the first time, a co-evolutionary mechanism between a solver and a questioner. This framework employs difficulty-aware sampling to dynamically identify medium-difficulty anchor problems aligned with the model’s current proficiency, generates diverse questions accordingly, and jointly optimizes both components for progressive reasoning improvement. By resolving issues of anchor-less generation, absent co-evolution, and difficulty misalignment, our method achieves state-of-the-art performance and strong generalization on mathematical and general reasoning benchmarks, using fewer than 2K real mathematical examples.

Abstract

Reinforcement learning (RL) has demonstrated potential for enhancing reasoning in large language models (LLMs). However, effective RL training, which requires medium-difficulty training samples, faces two fundamental challenges: Effective Data Scarcity and Dynamic Difficulty Shifts, where medium-difficulty samples are scarce and become trivial as models improve. Existing methods mitigate this scarcity to some extent by generating training samples. However, these approaches suffer from anchor-free generation, ignoring co-evolution, and difficulty mismatch. To address these issues, we propose D$^2$Evo, a Dual Difficulty-aware self-Evolution RL framework. In each iteration, our method mines medium-difficulty anchors based on the current Solver's capability, trains the Questioner to generate diverse questions at appropriate difficulty levels, and jointly optimizes both components to enable progressive reasoning gains. Extensive experiments demonstrate that D$^2$Evo outperforms existing methods on mathematical reasoning benchmarks with fewer than 2K real mathematical samples, and exhibits strong generalization on general reasoning benchmarks.

Problem

• Effective Data Scarcity
• Dynamic Difficulty Shifts
• Data-Efficient Reinforcement Learning
• Medium-Difficulty Samples

Innovation

• Dual Difficulty-Aware
• Self-Evolution
• Data-Efficient Reinforcement Learning
• Medium-Difficulty Anchors
• Co-Evolution