OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers

2026-07-02

AI Summary

Diffusion Transformers suffer from high inference costs in image and video generation, and existing post-training quantization (PTQ) methods rely on calibration data specific to particular timesteps, prompts, or modalities. This work proposes a data-agnostic, unified quantization framework that applies Randomly Permuted Block Hadamard (RPBH) rotations to map both weights and activations into a normalized rotational basis, where Lloyd-Max codebooks are employed for quantization. This approach enables a single shared codebook across all timesteps, prompts, and network layers, and facilitates seamless transfer from image to video generation without additional hyperparameter tuning. Evaluated on FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, the method achieves state-of-the-art performance in low-bit PTQ, delivering usable image generation quality for the first time under W2A4 (2-bit weights, 4-bit activations) settings.

Abstract

Diffusion transformers (DiTs) achieve state-of-the-art image and video generation, but their multi-step sampling and growing parameter count make inference expensive. Post-training quantization (PTQ) is the natural remedy, yet DiT activations shift across timesteps, prompts, and guidance branches, forcing prior methods to re-fit calibration data for every new checkpoint or modality. We present OrbitQuant, a data-agnostic weight-activation quantizer that bypasses range estimation by quantizing in a normalized, rotated basis. In this basis, a randomized permuted block-Hadamard (RPBH) rotation concentrates each coordinate around one fixed, known marginal regardless of the input, so a single Lloyd-Max codebook serves all timesteps, prompts, and layers of a given input dimension. We extend the same quantizer to weight rows offline, absorbing the rotation into the weights so that it cancels inside each linear layer and only a forward rotation on the activations remains at runtime. The same recipe transfers from image to video with no per-modality tuning. Across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, it sets the state of the art for PTQ at several low-bit settings. It also pushes PTQ of image diffusion transformers to W2A4 with usable generation quality.

Problem

• Diffusion Transformers
• Post-Training Quantization
• Activation Shift
• Data-Agnostic Quantization
• Inference Efficiency

Innovation

• data-agnostic quantization
• diffusion transformers
• post-training quantization
• rotated basis
• RPBH rotation