ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation

2024-06-04

arXiv.org

AI Summary

To address the high memory and computational overhead of diffusion Transformers (DiTs) in image and video generation—stemming from their large model size and multi-frame processing, which hinders deployment on edge devices—this paper presents the first systematic analysis of quantization error sources in DiTs and proposes ViDiT-Q, a generation-oriented quantization framework. ViDiT-Q supports post-training W8A8 and W4A8 quantization, integrating inter-layer sensitivity modeling, dynamic range calibration, and custom GPU kernel optimization. Evaluated across multiple DiT architectures, it achieves 2–2.5× memory compression and 1.4–1.7× end-to-end inference speedup, while preserving generation quality: degradation in FID and LPIPS remains below 1%, and visual fidelity is nearly lossless.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Learning on the Edge & Model Compression
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Diffusion transformers have demonstrated remarkable performance in visual generation tasks, such as generating realistic images or videos based on textual instructions. However, larger model sizes and multi-frame processing for video generation lead to increased computational and memory costs, posing challenges for practical deployment on edge devices. Post-Training Quantization (PTQ) is an effective method for reducing memory costs and computational complexity. When quantizing diffusion transformers, we find that existing quantization methods face challenges when applied to text-to-image and video tasks. To address these challenges, we begin by systematically analyzing the source of quantization error and conclude with the unique challenges posed by DiT quantization. Accordingly, we design an improved quantization scheme: ViDiT-Q (Video&Image Diffusion Transformer Quantization), tailored specifically for DiT models. We validate the effectiveness of ViDiT-Q across a variety of text-to-image and video models, achieving W8A8 and W4A8 with negligible degradation in visual quality and metrics. Additionally, we implement efficient GPU kernels to achieve practical 2-2.5x memory saving and a 1.4-1.7x end-to-end latency speedup.

Problem

• Reduce computational and memory costs
• Improve quantization for diffusion transformers
• Enable efficient deployment on edge devices

Innovation

• Quantization for diffusion transformers
• Efficient GPU kernel implementation
• Minimal quality degradation post-quantization