$R^2$-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction

2026-04-20

AI Summary

This work addresses the high latency in diffusion-based large language models (dLLMs) caused by spatial and temporal redundancies during the decoding phase. It presents the first systematic identification and modeling of such redundancies, introducing a unified redundancy reduction framework. At the inference level, the framework incorporates training-free decoding rules—specifically, local confidence aggregation and early fixation of stable tokens—to streamline generation. At the training level, it proposes a redundancy-aware supervised fine-tuning strategy that eliminates reliance on handcrafted thresholds. Experimental results demonstrate that the approach reduces decoding steps by up to 75% across diverse models and tasks while preserving generation quality.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to autoregressive generation by enabling parallel token prediction. However, practical dLLM decoding still suffers from high inference latency, which limits deployment. In this work, we observe that a substantial part of this inefficiency comes from recurring redundancy in the decoding process, including spatial redundancy caused by confidence clusters and positional ambiguity, and temporal redundancy caused by repeatedly remasking predictions that have already stabilized. Motivated by these patterns, we propose $R^2$-dLLM, a unified framework for reducing decoding redundancy from both inference and training perspectives. At inference time, we introduce training-free decoding rules that aggregate local confidence and token predictions, and finalize temporally stable tokens to avoid redundant decoding steps. We further propose a redundancy-aware supervised fine-tuning pipeline that aligns the model with efficient decoding trajectories and reduces reliance on manually tuned thresholds. Experiments demonstrate that $R^2$-dLLM consistently reduces the number of decoding steps by up to 75% compared to existing decoding strategies, while maintaining competitive generation quality across different models and tasks. These results validate that decoding redundancy is a central bottleneck in dLLMs, and that explicitly reducing it yields substantial practical efficiency gains.

Problem

• Diffusion Large Language Models
• inference latency
• spatial redundancy
• temporal redundancy
• decoding redundancy

Innovation

• diffusion LLM
• redundancy reduction
• parallel decoding
• efficient inference
• spatio-temporal redundancy