Simple Denoising Diffusion Language Models

2025-10-26

AI Summary

Existing discrete diffusion language models (e.g., MDLMs, USDMs) suffer from severe performance degradation under few-step generation and are incompatible with continuous-domain few-step distillation techniques. Although USDMs partially alleviate these limitations, their complex loss design hinders scalability. This paper proposes a simplified denoising training framework: it optimizes only over tokens replaced by noise, reformulates denoising as self-supervised learning, and introduces a contrastive-learning-inspired negative gradient mechanism to enhance discriminability. The approach significantly improves training stability and few-step generation quality, achieving a new state-of-the-art ELBO score. It outperforms existing discrete diffusion language models across multiple metrics, establishing a novel paradigm for efficient and scalable language generation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Diffusion models have recently been extended to language generation through Masked Diffusion Language Models (MDLMs), which achieve performance competitive with strong autoregressive models. However, MDLMs tend to degrade in the few-step regime and cannot directly adopt existing few-step distillation methods designed for continuous diffusion models, as they lack the intrinsic property of mapping from noise to data. Recent Uniform-state Diffusion Models (USDMs), initialized from a uniform prior, alleviate some limitations but still suffer from complex loss formulations that hinder scalability. In this work, we propose a simplified denoising-based loss for USDMs that optimizes only noise-replaced tokens, stabilizing training and matching ELBO-level performance. Furthermore, by framing denoising as self-supervised learning, we introduce a simple modification to our denoising loss with contrastive-inspired negative gradients, which is practical and yield additional improvements in generation quality.

Problem

• Improving few-step generation for discrete diffusion language models
• Simplifying complex loss formulations in uniform-state diffusion models
• Enhancing generation quality through contrastive-inspired denoising methods

Innovation

• Simplified denoising loss optimizes noise-replaced tokens
• Frames denoising as self-supervised learning process
• Uses contrastive-inspired negative gradients for quality