Improved Large Language Diffusion Models

2026-06-23

AI Summary

This work addresses the limitations of traditional autoregressive large language models, which rely on unidirectional attention and thus struggle to fully capture bidirectional context. To overcome this, the authors propose iLLaDA—the first 8B-parameter fully bidirectional masked diffusion language model trained from scratch. Both pretraining (on 12T tokens) and instruction tuning (on 25B tokens) employ a masked diffusion objective, augmented with variable-length generation and a confidence scoring mechanism to enhance efficiency and evaluation reliability. Experimental results demonstrate that iLLaDA-Base achieves gains of 21.6 and 14.9 points on BBH and ARC-Challenge, respectively, while iLLaDA-Instruct improves by 14.5 and 16.5 points on MATH and HumanEval. Overall, iLLaDA matches the performance of Qwen2.5 7B, validating the feasibility of diffusion-based paradigms for building high-performing non-autoregressive language models.

Abstract

Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present \emph{iLLaDA}, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.

Problem

• large language models
• masked diffusion
• bidirectional attention
• non-autoregressive training
• language model pre-training

Innovation

• masked diffusion language model
• bidirectional attention
• non-autoregressive generation
• variable-length generation
• confidence-based scoring