Veda: Scalable Video Diffusion via Distilled Sparse Attention

2026-05-28

AI Summary

This work addresses the computational bottleneck of diffusion Transformers in high-resolution long video generation, where self-attention incurs quadratic complexity and existing sparse attention methods suffer severe performance degradation under high sparsity. The authors propose Veda, a framework that formulates patch selection as an explicit reconstruction problem from full attention, revealing that generation quality hinges on the geometric alignment between the sparse mask and full attention patterns rather than sparsity ratio alone. Building on this insight, they introduce a statistically aware patch scoring mechanism, a head-aware partitioning strategy, and a hardware-efficient patch-skipping kernel to construct a scalable distilled sparse attention. Evaluated on Waver-T2V-12B for 720p 10-second video generation, Veda achieves a 5.1× end-to-end speedup (10.5× in self-attention), reduces attention overhead from 92% to 50%, and exhibits increasingly pronounced acceleration with longer sequences.

Abstract

Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1$\times$ end-to-end speedup and a 10.5$\times$ self-attention speedup, reducing attention overhead from 92% to 50%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.

Problem

• video diffusion
• scalability
• sparse attention
• self-attention
• high-resolution video generation

Innovation

• distilled sparse attention
• tile-wise geometry
• video diffusion
• hardware-efficient kernel
• scalable generation