Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning

2026-05-21

AI Summary

This work addresses a key limitation of conventional Online Policy Self-Distillation (OPSD), which employs a uniform supervision direction that suppresses predictive uncertainty in large language models during complex reasoning, thereby impairing their capacity for exploration and hypothesis refinement. To overcome this, the paper introduces Direction-Adaptive Self-Distillation (DASD), the first method to leverage token-level entropy as a routing signal for supervision direction: high-entropy tokens are steered away from the teacher model to preserve exploratory behavior, while low-entropy tokens are drawn closer to enhance execution stability. Through entropy-aware directional self-distillation and token-level supervision modulation, DASD achieves state-of-the-art macro Avg@16 performance across six mathematical reasoning benchmarks, with comprehensive evaluations—including Pass@$k$, reasoning health metrics, and generalization analyses—confirming its effectiveness.

Abstract

On-policy self-distillation (OPSD) is an emerging LLM post-training paradigm in which the model serves as its own teacher: conditioned on privileged information such as a reference trace or hint, the same policy provides dense token-level supervision on its own rollouts. However, recent studies show that OPSD degrades complex reasoning by suppressing predictive uncertainty, which supports exploration and hypothesis revision. Our token-level analysis shows that this failure arises from applying a uniform direction of teacher supervision across tokens with different uncertainty levels: conformity to the privileged self-teacher suppresses exploration at high entropy, while deviation from the teacher degrades step accuracy at low entropy. Accordingly, we propose \textbf{Direction-Adaptive Self-Distillation} (\textbf{DASD}), which reframes privileged self-distillation from uniform teacher imitation into entropy-routed directional supervision: high-entropy tokens are pushed away from the privileged teacher to preserve exploration, while low-entropy tokens are pulled toward the teacher to stabilize step-level execution. Across six mathematical reasoning benchmarks, DASD achieves the best macro Avg@16 over strong RLVR and self-distillation baselines. Pass@$k$, reasoning-health, and generalization analyses show that these average gains come from preserving exploration without sacrificing step-level execution.

Problem

• self-distillation
• LLM reasoning
• predictive uncertainty
• token-level supervision
• exploration-exploitation trade-off

Innovation

• Direction-Adaptive Self-Distillation
• on-policy self-distillation
• predictive uncertainty
• entropy-aware supervision
• LLM reasoning