DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention

2026-03-09

AI Summary

Diffusion language models suffer from inefficient inference due to redundant recomputation of the entire sequence during iterative denoising. This work proposes a training-free acceleration framework that, for the first time, exploits the temporal sparsity of token representations across diffusion steps. By dynamically identifying salient tokens via cosine similarity of attention contexts, the method performs full feed-forward and attention computations only on these tokens, while reusing cached activation values for the rest. Integrating saliency-based token selection, partial attention mechanisms, and activation caching, the approach achieves up to 9.6× higher throughput on multiple reasoning and code generation benchmarks, while largely preserving the original accuracy of state-of-the-art models such as LLaDA and Dream.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Masked Diffusion Language Models (MDLMs) enable parallel token decoding, providing a promising alternative to the sequential nature of autoregressive generation. However, their iterative denoising process remains computationally expensive because it repeatedly processes the entire sequence at every step. We observe that across these diffusion steps, most token representations remain stable; only a small subset, which we term salient tokens, contributes meaningfully to the next update. Leveraging this temporal sparsity, we present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only these salient tokens. DyLLM identifies saliency by measuring the cosine similarity of attention contexts between adjacent denoising steps. It recomputes feed-forward and attention operations only for salient tokens while reusing cached activations for the remainder. Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput while largely preserving the baseline accuracy of state-of-the-art models like LLaDA and Dream.

Problem

• Diffusion Language Models
• Inference Efficiency
• Token Selection
• Computational Cost
• Parallel Decoding

Innovation

• Diffusion Language Models
• Token Saliency
• Efficient Inference
• Partial Attention
• Temporal Sparsity