ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

2026-06-22

AI Summary

Video Diffusion Transformers suffer from prohibitive computational and memory costs due to their full 3D attention mechanism, and existing sparsification approaches struggle to balance efficiency with generation quality. This work proposes a training-free sparse attention framework that, for the first time, reveals how attention heads rapidly converge during inference to input-agnostic, intrinsically sparse topologies. Building on this insight, the authors design a scale-invariant, hardware-friendly block-sparse mechanism that integrates WEST (Weight-encoded Sparse Topology) for offline extraction of stable attention masks and FAST (Fidelity-Aware Sensitivity Tuning) for adaptive per-head sparsity calibration. Together, these components enable co-optimization with bit-level hardware-aligned kernels. Evaluated on Wan2.1, the method achieves up to 1.90× end-to-end speedup while maintaining or even enhancing generation fidelity, significantly outperforming current state-of-the-art approaches.

Abstract

While Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, their reliance on 3D full attention creates a quadratic computational bottleneck. Existing sparse methods face a dilemma: dynamic pruning suffers from prohibitive runtime overhead and memory fragmentation, while static heuristics fail to capture fine-grained dependencies. In this work, we propose ScalingAttention, a training-free framework grounded in a key inductive bias: while individual activations are input-dependent, the high-mass attention regions for each head rapidly converge to a stable, prompt-agnostic Intrinsic Sparse Topology. This topology is weight-encoded, scale-invariant, and efficient to extract. ScalingAttention decouples topology discovery from sparsity control via: (1) WEST (Weight-Encoded Sparse Topology), which extracts a robust block-sparse prior mask offline to eliminate runtime search; (2) FAST (Fidelity-Aware Sensitivity Tuning), which adaptively tunes head-wise sparsity based on diffusion fidelity requirements. To ensure practical acceleration, we co-design a hardware-aligned bit-wise block-sparse kernel. Experiments on Wan2.1 show up to 1.90X end-to-end speedup with superior fidelity, establishing a new Pareto frontier over state-of-the-art baselines.

Problem

• Diffusion Transformers
• Sparse Attention
• Computational Bottleneck
• Video Generation
• Attention Topology

Innovation

• Intrinsic Sparse Topology
• Weight-Encoded Sparse Topology
• Fidelity-Aware Sensitivity Tuning
• Block-Sparse Attention
• Training-Free Sparsification