DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training

2026-06-29

AI Summary

This work addresses the challenge that large language models struggle to stably self-improve without external supervision, as existing methods often lack dynamic awareness of problem difficulty and consequently over-optimize on easy samples while neglecting hard or boundary cases. To overcome this, the paper proposes DRIFT, a novel framework that introduces, for the first time, a synergistic mechanism combining problem-level difficulty routing with token-level pacing gating. This mechanism dynamically allocates self-distillation and reinforcement learning signals and guides exploration at critical reasoning positions. DRIFT further incorporates a success experience buffer and a two-stage curriculum learning strategy to enable fine-grained control over the self-improvement process. Evaluated across five benchmarks and three model scales, DRIFT significantly outperforms GRPO and SDPO, achieving an average score of 79.5% and a ToolUse task accuracy of 79.2%, establishing new state-of-the-art results.

Abstract

Enabling large language models to achieve stable self-improvement without external expert supervision remains a central challenge in complex reasoning tasks. Existing self-distillation and reinforcement learning methods lack explicit mechanisms for tracking problem-level learning progress and adapting optimization strategies accordingly. Consequently, training may over-optimize easy problems, receive weak supervision from hard problems, and fail to sufficiently explore borderline cases. To resolve these issues, we propose DRIFT, an online self-evolution policy optimization framework for large language models. DRIFT regulates the model's self-improvement process through the joint use of Difficulty Routing and Rhythm Gating. The former identifies the model's learning state at the problem level and dynamically allocates self-distillation and reinforcement learning signals, while the latter refines policy updates at the token level, concentrating exploration on critical reasoning positions. By further incorporating a success buffer and a two-stage curriculum learning strategy, DRIFT preserves high-quality historical experience while progressively guiding the model from reliable behavior acquisition toward stable policy evolution. Evaluated across five benchmarks and three model scales, DRIFT surpasses the peak performance of both GRPO and SDPO across all evaluated metrics. On the average score over the five benchmarks, DRIFT achieves 79.5$\%$, outperforming GRPO by 9.5$\%$ and SDPO by 7.5$\%$, establishing a new state-of-the-art result. Notably, on ToolUse, DRIFT reaches an accuracy of 79.2$\%$, improving over GRPO by 13.5$\%$ and SDPO by 10.7$\%$, setting a new state-of-the-art and substantially outperforming all concurrent methods.

Problem

• self-improvement
• complex reasoning
• self-distillation
• reinforcement learning
• learning progress tracking

Innovation

• Difficulty Routing
• Rhythm Gating
• Self-Distillation
• Success Buffer
• Policy Optimization