A Brief Overview: On-Policy Self-Distillation In Large Language Models

2026-05-18

AI Summary

This work proposes On-Policy Self-Distillation (OPSD), a framework that eliminates the need for external teacher models in knowledge distillation by enabling a single language model to act as both teacher and student. The teacher generates outputs based on actual reasoning trajectories, while the student predicts solely from problem descriptions; self-alignment is achieved by minimizing the divergence between their output distributions. By directly leveraging ground-truth solution information without relying on off-policy teachers, OPSD effectively mitigates distributional mismatch inherent in conventional distillation approaches. Experimental results demonstrate that, compared to standard on-policy distillation, the proposed method maintains reasoning consistency while reducing GPU memory consumption by 40%–60%, substantially enhancing training efficiency.

Abstract

On-Policy Self-Distillation (OPSD) introduces a unified learning framework in which a single large language model simultaneously serves as both teacher and student. Unlike conventional knowledge distillation that relies on a separate, often larger teacher model, OPSD operates under different contextual roles: the teacher policy is granted privileged access to verified reasoning traces, while the student policy observes only the problem statement. OPSD is trained to minimize per-token distributional divergence between the two roles over trajectories sampled from the student itself, thereby aligning its own reasoning behavior with solution-aware rationalizations. OPSD eliminates the need for an external teacher, directly leverages ground-truth solution information, and resolves the distribution mismatch inherent in off-policy distillation. OPSD typically reduces GPU memory consumption by approximately 40%-60% compared to standard On-Policy Distillation (OPD). In this paper, we present a brief analysis of the conceptual foundations, methodological innovations, and principled designs underlying recent advances in OPSD for large language models. This discussion, crafted from the perspective of beginners in this field, aims to provide a concise overview of the design principles and emerging patterns of OPSD in LLMs, intended for researchers who are similarly new to this area.

Problem

• On-Policy Self-Distillation
• Large Language Models
• Knowledge Distillation
• Distribution Mismatch
• Teacher-Free Learning

Innovation

• On-Policy Self-Distillation
• Large Language Models
• Knowledge Distillation
• Reasoning Alignment
• Memory Efficiency