GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

2026-05-28

AI Summary

This work addresses the performance degradation in diffusion language models caused by training–inference mismatch when optimizing denoisers via reinforcement learning (RL), where intractable policy likelihoods necessitate the use of ELBO surrogates. To overcome this limitation, the authors propose Guided Denoiser Self-Distillation (GDSD), which reformulates RL for diffusion language models as a normalization-free self-distillation process. GDSD constructs an advantage-guided self-teacher based on the closed-form optimal solution of reverse-KL regularized RL and directly distills denoiser logits, thereby circumventing the bias introduced by ELBO-based proxies. Experiments on LLaDA-8B and Dream-7B demonstrate that GDSD substantially outperforms existing methods across planning, mathematical reasoning, and code generation tasks, achieving up to a 19.6% improvement in test accuracy while exhibiting more stable training reward dynamics.

Abstract

Reinforcement learning (RL) can be used to improve the policy (denoiser) of diffusion large language models (dLLMs), while being hindered by the intractability of the policy likelihood. A dominant and efficient family of methods replaces the likelihood in standard RL with its evidence lower bound (ELBO), estimated from randomly masked sequences. Despite being well aligned with pre-training, these approaches introduce bias through training--inference mismatch by using the ELBO as a likelihood surrogate, which can degrade performance. In this work, we propose Guided Denoiser Self-Distillation (GDSD) to directly distill the denoiser of dLLMs from an advantage-guided self-teacher, derived from the closed-form optimum of reverse-KL regularized RL. GDSD matches the dLLM's denoiser logits to the teacher's via a normalization-free objective, which reduces RL to likelihood-free self-distillation and thus bypasses the TIM biases. Recent ELBO-based methods emerge as instances of applying different distillation divergences, but with diagnosable pathologies that GDSD avoids. On planning, math, and coding benchmarks with LLaDA-8B and Dream-7B, GDSD consistently outperforms prior state-of-the-art ELBO-based methods with a more stable training reward dynamics, achieving test-accuracy improvements of up to $+19.6\%$. These results suggest that direct denoiser self-distillation, without relying on an ELBO likelihood surrogate, can provide a more stable and effective RL procedure for dLLMs. Code is available at https://github.com/GaryBall/GDSD.

Problem

• Reinforcement Learning
• Diffusion Language Models
• Training-Inference Mismatch
• ELBO
• Policy Likelihood

Innovation

• self-distillation
• diffusion language models
• reinforcement learning
• training-inference mismatch
• ELBO-free