LiteAttention: A Temporal Sparse Attention for Diffusion Transformers

2025-11-14

AI Summary

Video diffusion Transformers suffer from high latency due to quadratic attention complexity. Existing acceleration methods face a fundamental trade-off: dynamic sparsity incurs substantial computational overhead and accuracy degradation, while static sparsity suffers from fixed, suboptimal patterns. This work is the first to empirically identify and exploit the strong temporal coherence of attention sparsity patterns across denoising timesteps in diffusion models. We propose an evolutionary stride-based computation skipping mechanism: built upon FlashAttention, our optimized kernel identifies non-critical token tiles early and propagates skip decisions forward, enabling fine-grained dynamic sparsity without repeated sparsity analysis. The method uniquely combines dynamic adaptability with static efficiency. Evaluated on production-scale video diffusion models, it significantly reduces computational cost—up to 42% FLOPs reduction—while preserving generation quality, demonstrating strong practical deployability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Diffusion Transformers, particularly for video generation, achieve remarkable quality but suffer from quadratic attention complexity, leading to prohibitive latency. Existing acceleration methods face a fundamental trade-off: dynamically estimating sparse attention patterns at each denoising step incurs high computational overhead and estimation errors, while static sparsity patterns remain fixed and often suboptimal throughout denoising. We identify a key structural property of diffusion attention, namely, its sparsity patterns exhibit strong temporal coherence across denoising steps. Tiles deemed non-essential at step $t$ typically remain so at step $t+delta$. Leveraging this observation, we introduce LiteAttention, a method that exploits temporal coherence to enable evolutionary computation skips across the denoising sequence. By marking non-essential tiles early and propagating skip decisions forward, LiteAttention eliminates redundant attention computations without repeated profiling overheads, combining the adaptivity of dynamic methods with the efficiency of static ones. We implement a highly optimized LiteAttention kernel on top of FlashAttention and demonstrate substantial speedups on production video diffusion models, with no degradation in quality. The code and implementation details will be publicly released.

Problem

• Reduces quadratic attention complexity in diffusion transformers for video generation
• Overcomes trade-off between dynamic sparse attention overhead and static pattern rigidity
• Exploits temporal coherence of attention patterns to eliminate redundant computations

Innovation

• Leverages temporal coherence across denoising steps
• Marks non-essential tiles early to skip computations
• Combines dynamic adaptivity with static efficiency