Rethinking On-Policy Self-Distillation for Thinking Models

2026-07-06

AI Summary

This work investigates the adverse effects of privileged information self-distillation on strong reasoning models, revealing that it degrades performance—particularly along long reasoning trajectories—by impairing self-correction and backtracking capabilities at high-entropy branching points. Through systematic analysis of in-policy distillation (OPD), we demonstrate for the first time that this approach significantly suppresses corrective behaviors and reduces reasoning diversity. Leveraging trajectory-level diagnostics, high-entropy fork identification, and length-normalized evaluation across five models—including Qwen3 and OLMo—we observe up to a 17% drop in avg@16 accuracy on AIME24/25 and HMMT25 benchmarks. Moreover, key token-level signals such as verification, backtracking, and hesitation are markedly diminished. Our findings underscore the necessity of fine-grained scrutiny of reasoning dynamics to design more effective distillation strategies.

Abstract

Self-distillation is a promising recipe for self-improvement in language models. In this setting, a model can serve as its own teacher when given privileged information, such as a solution to a math problem. This seems especially appealing for thinking models, which can use test-time reasoning to absorb the privileged information. Surprisingly, we show that privileged self-distillation degrades thinking models on long reasoning traces: across five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy. The degradation scales with the amount of privileged context withheld from the student and is most pronounced at long rollout budgets, where thinking models otherwise obtain their largest gains. This failure mode is not specific to self-distillation: on-policy distillation (OPD) improves thinking models, but privileged OPD reverses these gains. Our diagnostics link this failure mode to how privileged teacher context reshapes learning at high-entropy forking positions, where multiple continuations remain plausible and may lead to different reasoning paths. Privileged context lowers fork rates in thinking-model rollouts but not in instruction-model rollouts. This leads to an interesting dichotomy, where privileged context can help instruction-tuned models but hurts stronger thinking models. The effect is visible when the student begins a self-correction branch, where privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports. Thinking models trained with a privileged teacher produce fewer verification, backtracking, and hedging markers, even after length normalization. These findings indicate that self-distillation for strong thinking models requires attention to token-level signal, especially around correction and reasoning steps.

Problem

• self-distillation
• thinking models
• privileged context
• on-policy distillation
• reasoning traces

Innovation

• privileged self-distillation
• thinking models
• on-policy distillation
• reasoning traces
• forking positions