Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design

2026-06-13

AI Summary

This work addresses the challenges in diffusion language models (DLMs), whose performance is highly sensitive to the design of the transition kernel, often leading to training instability, slow convergence, and sampling bias. By analyzing asymptotic bias, exposure bias, and optimization variance in generalization error, the authors propose SemDLM+, a method that performs diffusion within semantically similar neighborhoods to effectively balance posterior approximation difficulty and sampling correction capability. The approach elucidates the role of different diffusion kernels in the bias–variance trade-off and integrates global transitions with semantic frequency penalization to mitigate the semantic basin problem. Experiments on LM1B and OpenWebText demonstrate that SemDLM+ significantly improves training dynamics and enhances generation diversity while maintaining competitive language modeling performance.

Abstract

Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models. However, their performance is highly sensitive to the choice of transition kernels, and poorly designed kernels can lead to issues like training instability, slow convergence, and biased sampling. In this paper, we study this sensitivity through a principled analysis of generalization error and identify three critical factors: asymptotic bias (difficulty in approximating the posterior distribution), exposure bias (error propagation during sampling), and optimization variance induced by kernel dispersion. We further compare different transition kernels: masking diffusion yields sparse and easier posterior-approximation targets, while uniform diffusion provides stronger sampling-side repair but induces harder approximation. Motivated by this trade-off, we revisit a previously overlooked variant, semantic DLM (SemDLM), where the transition kernel corrupts tokens to neighborhoods that are semantically similar. Our theory suggests that SemDLM can serve as a plausible middle ground by reducing the posterior approximation difficulty of uniform diffusion while retaining repair ability. However, we find that SemDLM suffers from a semantic basin problem, where sampling repeatedly stays within a semantic region and produces low-diversity text. To address this, we propose SemDLM+, which adds a global transition and a semantic-frequency penalty during sampling. Experiments on LM1B and OpenWebText show that SemDLM+ improves training dynamics and achieves competitive language modeling and generation quality with satisfactory diversity.

Problem

• Diffusion Language Models
• transition kernel
• semantic basin
• sampling bias
• posterior approximation

Innovation

• Diffusion Language Models
• Transition Kernel
• Bias-variance Trade-off
• Semantic Diffusion
• Sampling Diversity