Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs

2025-08-20

AI Summary

Diffusion large language models (dLLMs) pose significant challenges for efficient deployment on resource-constrained edge devices due to their massive parameter counts and activation tensors containing prominent outliers. This work presents the first systematic study of post-training quantization (PTQ) for dLLMs, identifying outlier-dominated activation dynamic ranges as the primary bottleneck for accuracy degradation under low-bit quantization. We conduct comprehensive empirical evaluations across multiple tasks, model variants, bitwidths, quantization schemes, task types, and architectural configurations. Based on these findings, we propose targeted PTQ optimizations specifically tailored to dLLM characteristics. Experimental results demonstrate that our approach substantially improves quantization accuracy at 4–6 bits—achieving performance close to full-precision in several configurations. To foster reproducibility and adoption, we publicly release all code and configuration files. This work provides both theoretical insights and practical guidelines for lightweight deployment of non-autoregressive generative models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in diffusion large language models (dLLMs) have introduced a promising alternative to autoregressive (AR) LLMs for natural language generation tasks, leveraging full attention and denoising-based decoding strategies. However, the deployment of these models on edge devices remains challenging due to their massive parameter scale and high resource demands. While post-training quantization (PTQ) has emerged as a widely adopted technique for compressing AR LLMs, its applicability to dLLMs remains largely unexplored. In this work, we present the first systematic study on quantizing diffusion-based language models. We begin by identifying the presence of activation outliers, characterized by abnormally large activation values that dominate the dynamic range. These outliers pose a key challenge to low-bit quantization, as they make it difficult to preserve precision for the majority of values. More importantly, we implement state-of-the-art PTQ methods and conduct a comprehensive evaluation across multiple task types and model variants. Our analysis is structured along four key dimensions: bit-width, quantization method, task category, and model type. Through this multi-perspective evaluation, we offer practical insights into the quantization behavior of dLLMs under different configurations. We hope our findings provide a foundation for future research in efficient dLLM deployment. All codes and experimental setups will be released to support the community.

Problem

• Studying quantization challenges for diffusion large language models
• Identifying activation outliers that hinder low-bit precision
• Evaluating PTQ methods across tasks and model variants

Innovation

• Systematic study of post-training quantization
• Identifying activation outliers for precision preservation
• Comprehensive multi-perspective evaluation framework