Training-free sparse attention based on cumulative energy filtering

2026-06-15

AI Summary

Existing sparse attention methods in video generation diffusion Transformers struggle to maintain generation quality under high sparsity. This work formulates token filtering as a bi-objective optimization problem balancing sparsity and fidelity, and introduces a training-free dynamic thresholding mechanism that adaptively selects tokens based on cumulative energy to maximize sparsity under a fixed recall constraint. The approach integrates seamlessly into Flash Attention without requiring additional mask computation. Experiments on Wan 2.2 demonstrate that the method increases sparsity from 61.42% to 82%, with less than a 5% drop in VBench scores, reduces attention computation by 15%, and achieves a 1.61× speedup over the baseline—significantly outperforming BLASST, which attains only a 1.18× speedup.

Abstract

Sparse attention accelerates Diffusion Transformers (DiTs) for video generation by computing only the important tokens while skipping the rest. The token selection strategy is key to balancing sparsity and accuracy. We formulate the token filtering process as a dual-goal optimization problem: maximizing sparsity and minimizing accuracy degradation. Existing algorithms cannot fulfill both objectives simultaneously. For example, Top-p only considers the accuracy constraint, while Top-k maintains a fixed computational budget but loosens the accuracy constraint. This paper demonstrates that maintaining a fixed recall rate is sufficient for ensuring accuracy, whereas a fixed threshold is suboptimal for reducing computational cost. Therefore, we propose a dynamic thresholding scheme to improve sparsity while maintaining the same level of accuracy. Furthermore, our algorithm is deeply integrated with Flash Attention (FA), eliminating the need for any additional masking computation overhead. Experimental results on Wan 2.2 validate that, compared to the BLASST algorithm which is also integrated with FA, our dynamic thresholding strategy enhances sparsity from 61.42\% to 82\% with a VBench metric drop of less than 5\%. This results in an approximate 15\% in attention computation and a $1.61\times$ increase in computational efficiency, which is 1.18x higher than that of BLASST.

Problem

• sparse attention
• diffusion transformers
• token selection
• computational efficiency
• accuracy-sparsity trade-off

Innovation

• sparse attention
• dynamic thresholding
• cumulative energy filtering
• Diffusion Transformers
• Flash Attention