LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration

2026-08-12

AI Summary

This work addresses the quadratic computational complexity of self-attention in video diffusion models, which hinders high-resolution and long-duration generation. The authors propose a training-free dynamic sparse attention method that substantially reduces computational overhead while preserving near-lossless attention fidelity—retaining 99% of original quality. Key innovations include block-wise precise attention quality estimation, dynamic selection of critical key/value blocks, cross-denoising-step reuse of frozen indices, and synergistic optimization with feature caching. Experiments demonstrate a 1.36× speedup on Wan2.1-1.3B with only a 0.06-point drop in VBench score; when combined with feature caching on HunyuanVideo, the approach achieves a 3.2× acceleration with merely a 0.02-point quality degradation, significantly outperforming existing baselines.

Abstract

Video diffusion transformers are costly to sample: every denoising step applies self-attention over a long 3D token sequence, a quadratic cost that dominates as resolution and duration grow. Sparse attention reduces this cost without retraining, but existing methods pursue aggressive sparsity, where further speedup costs disproportionately more attention fidelity. We target the opposite end of this trade-off: fix near-lossless fidelity by construction, and remove as much computation as this constraint permits. Two observations make this regime practical: roughly 40% of block interactions can be removed while retaining 99% of the attention mass, and the high-mass support remains stable across denoising steps. We propose LoSA, a training-free sparse-attention method that fixes a retained-mass threshold of 99% rather than a sparsity ratio: it measures exact block attention masses at one early dense step, keeps, for each head and query block, the smallest key/value block set meeting the threshold, and reuses the frozen block indices for all remaining steps. On Wan2.1-1.3B, LoSA alone gives a $1.36\times$ speedup with a 0.06-point VBench Overall drop. The benefit is largest under composition: combined with feature caching, LoSA reaches a $3.2\times$ speedup on HunyuanVideo at a 0.02-point drop, versus 0.32 points for the strongest sparse baseline at comparable speed. Across three video diffusion transformers and speedups up to $3.2\times$, LoSA consistently achieves the best training-free speed-quality trade-off.

Problem

• video diffusion
• sparse attention
• computational cost
• attention fidelity
• training-free acceleration

Innovation

• sparse attention
• training-free acceleration
• video diffusion models
• near-lossless fidelity
• attention sparsity