RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

2026-06-30

AI Summary

This work addresses the computational bottleneck in DiT-based video generation models caused by the quadratic complexity of 3D Rotary Position Embedding (RoPE) attention with respect to sequence length, compounded by the incompatibility of existing INT4 quantization methods with RoPE and their inefficiency in quantizing non-negative attention matrices. To overcome these challenges, the authors propose RotateAttention, a mixed-precision INT4 FlashAttention framework tailored for 3D RoPE DiTs. Their approach introduces a RoPE-aware rotation matrix to suppress outliers in query and key activations and devises a range-optimized asymmetric quantization strategy for the attention P-matrix, enabling effective synergy between RoPE and low-bit quantization for the first time. Combined with a selective FP16 fallback mechanism, this method achieves up to 1.68× end-to-end speedup and 2.2× kernel-level acceleration with negligible degradation in generation quality.

Abstract

In \textbf{DiT-based video generation models equipped with 3D Rotary Position Embeddings (3D RoPE)}, the attention mechanism remains a primary computational bottleneck due to its quadratic complexity with respect to sequence length. While quantized \textbf{FlashAttention} offers a promising path toward hardware acceleration, existing low-bit quantization methods overlook two critical challenges in this setting: \textbf{1)} applying online rotation matrices -- a widely used technique for mitigating outliers in Queries ($Q$) and Keys ($K$) -- is difficult to reconcile with \textbf{RoPE}; and \textbf{2)} the non-negative attention matrix $P = \exp(QK - \max(QK))$ makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributions of $Q$ and $K$ are strongly affected by the dimensional partitioning of \textbf{3D RoPE}. Based on this finding, we propose \textbf{RotateAttention}, an efficient \textbf{mixed-precision INT4 FlashAttention} framework tailored for \textbf{DiT-based video generation models with 3D RoPE}, using selective \textbf{FP16 fallback} for accuracy-sensitive attention blocks and denoising steps. RotateAttention introduces two core techniques: \textbf{1) RoPE-aware Rotation}, which employs either mergeable rotation matrices that can be fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in $Q$ and $K$; and \textbf{2) Range-optimized $P$ Quantization}, which uses fixed scales and zero-points to fully exploit the \textbf{INT4 numerical range} with minimal computational overhead. Experiments show that \textbf{RotateAttention} preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68$\times$ end-to-end speedup and 2.2$\times$ kernel-level acceleration.

Problem

• 3D RoPE
• INT4 quantization
• attention mechanism
• outlier mitigation
• dynamic range utilization

Innovation

• RotateAttention
• INT4 Quantization
• 3D RoPE
• FlashAttention
• Mixed-Precision