Problem
Research questions and friction points this paper is trying to address.
on-policy self-distillation
diffusion LLMs
arbitrary-order generation
post-training
token-level supervision
Learning from the Self-future: On-policy Self-distillation for dLLMs
🤖 AI Summary
Existing on-policy self-distillation methods are tailored for autoregressive models and struggle to accommodate the arbitrary-order generation capability inherent in diffusion-based large language models (dLLMs). This work proposes d-OPSD, the first framework to extend on-policy self-distillation to dLLMs. It introduces a “self-teacher” conditioned on model-generated completions as suffixes and aligns the iterative denoising process through step-level supervision, thereby overcoming the limitations of conventional prefix conditioning and token-level supervision. Evaluated on four reasoning benchmarks, d-OPSD significantly outperforms RLVR and supervised fine-tuning baselines, achieving superior performance with only about 10% of the optimization steps, thus substantially improving sample efficiency.
📝 Abstract
On-policy self-distillation (OPSD) has proven effective for post-training large language models (LLMs), yet its application to diffusion LLMs (dLLMs) remains unexplored. Existing OPSD methods are inherently autoregressive-centric. They inject privileged information via left-to-right prefix conditioning with token-level divergence supervision, a design that fundamentally conflicts with the arbitraryorder generation of dLLMs. We introduce d-OPSD, the first OPSD framework tailored for dLLMs. Our approach makes two core contributions. First, we reframe self-teacher construction by using self-generated answers as suffix conditioning, enabling the student model to learn from "self future-experience" rather than privileged prefixes. Second, we shift supervision from token-level to step-level, aligning training with the iterative denoising process of dLLMs. Experiments across four reasoning benchmarks show that d-OPSD consistently outperforms RLVR and SFT baselines with superior sample efficiency, requiring only around 10% of the optimization steps by RLVR and opening a promising pathway for dLLM posttraining. The code is available at https://github.com/xingzhejun/d-OPSD.
Innovation
Methods, ideas, or system contributions that make the work stand out.
on-policy self-distillation
diffusion LLMs
suffix conditioning
step-level supervision
sample efficiency
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
