CDLM: Consistency Diffusion Language Models For Faster Sampling

2025-11-24

AI Summary

Diffusion language models (DLMs) suffer from low inference efficiency due to their multi-step refinement process and incompatibility with key-value (KV) caching. This paper introduces CDLM—the first training-based acceleration method for DLMs that fully supports standard KV caching. CDLM enables multi-token parallel generation via consistency modeling and introduces a block-wise causal attention mechanism, preserving strict causality while achieving full compatibility with conventional KV caches. Built upon standard Transformer architectures, CDLM requires only lightweight fine-tuning, incorporating a consistency diffusion objective, block-wise causal masking, and parallel decoding strategies. On mathematical reasoning and code generation benchmarks, CDLM reduces inference latency by 3.6–14.5× over baseline DLMs, maintaining accuracy on par with the original models. To our knowledge, CDLM is the first approach to unify efficient KV cache utilization with high-quality, few-step generation in diffusion-based language modeling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Diffusion Language Models (DLMs) offer a promising parallel generation paradigm but suffer from slow inference due to numerous refinement steps and the inability to use standard KV caching. We introduce CDLM (Consistency Diffusion Language Models), a training-based acceleration method that simultaneously tackles both bottlenecks. CDLM integrates consistency modeling to drastically reduce the number of required sampling steps by enabling multi-token finalization. Furthermore, we enforce a block-wise causal attention mask during fine-tuning, making the model fully compatible with KV caching. Experiments show CDLM achieves 3.6x-14.5x lower latency while maintaining competitive accuracy on math and coding tasks. The full training and evaluation code is available at https://github.com/SqueezeAILab/CDLM.

Problem

• Accelerates Diffusion Language Models by reducing sampling steps
• Enables KV caching compatibility through causal attention masking
• Maintains competitive accuracy while achieving significant latency reduction

Innovation

• Integrates consistency modeling for faster sampling
• Uses block-wise causal attention mask for KV caching
• Enables multi-token finalization with reduced steps