LLaDA2.1: Speeding Up Text Diffusion via Token Editing

2026-02-09

AI Summary

This work addresses the inherent trade-off between decoding speed and generation quality in large language diffusion models by proposing a dual-mode, configurable-threshold decoding architecture that integrates a Token-to-Token (T2T) editing mechanism with a Mask-to-Token (M2T) framework, enabling both high-speed and high-quality generation modes. It introduces, for the first time in diffusion language models, a large-scale reinforcement learning (RL) alignment mechanism, enhanced by an extended context window and an RL framework specifically designed for diffusion-based large language models (dLLMs), substantially improving instruction following and reasoning capabilities. The authors release two models—LLaDA2.1-Mini (16B) and LLaDA2.1-Flash (100B)—which achieve strong performance across 33 benchmarks, with LLaDA2.1-Flash attaining a HumanEval+ decoding speed of 892 tokens per second (TPS).

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While LLaDA2.0 showcased the scaling potential of 100B-level block-diffusion models and their inherent parallelization, the delicate equilibrium between decoding speed and generation quality has remained an elusive frontier. Today, we unveil LLaDA2.1, a paradigm shift designed to transcend this trade-off. By seamlessly weaving Token-to-Token (T2T) editing into the conventional Mask-to-Token (M2T) scheme, we introduce a joint, configurable threshold-decoding scheme. This structural innovation gives rise to two distinct personas: the Speedy Mode (S Mode), which audaciously lowers the M2T threshold to bypass traditional constraints while relying on T2T to refine the output; and the Quality Mode (Q Mode), which leans into conservative thresholds to secure superior benchmark performances with manageable efficiency degrade. Furthering this evolution, underpinned by an expansive context window, we implement the first large-scale Reinforcement Learning (RL) framework specifically tailored for dLLMs, anchored by specialized techniques for stable gradient estimation. This alignment not only sharpens reasoning precision but also elevates instruction-following fidelity, bridging the chasm between diffusion dynamics and complex human intent. We culminate this work by releasing LLaDA2.1-Mini (16B) and LLaDA2.1-Flash (100B). Across 33 rigorous benchmarks, LLaDA2.1 delivers strong task performance and lightning-fast decoding speed. Despite its 100B volume, on coding tasks it attains an astounding 892 TPS on HumanEval+, 801 TPS on BigCodeBench, and 663 TPS on LiveCodeBench.

Problem

• decoding speed
• generation quality
• text diffusion
• large language models
• speed-quality trade-off

Innovation

• Token-to-Token editing
• threshold-decoding scheme
• diffusion language models
• reinforcement learning for dLLMs
• high-throughput text generation