Accelerating Text-to-Video Generation with Calibrated Sparse Attention

2026-03-05

AI Summary

Text-to-video diffusion models suffer from slow inference due to the high computational complexity of spatiotemporal attention. This work proposes CalibAtt, a training-free method that leverages offline calibration to identify block-level sparse and redundant attention patterns stable across inputs. By constructing layer-, head-, and diffusion-step-specific sparse attention operators, CalibAtt dynamically skips redundant connections during inference. It is the first approach to exploit query-stable block sparsity for hardware-friendly acceleration without requiring model fine-tuning. Evaluated on large-scale models such as Wan 2.1 14B and Mochi 1, CalibAtt achieves up to 1.58× end-to-end speedup while preserving generation quality and text-video alignment fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Recent diffusion models enable high-quality video generation, but suffer from slow runtimes. The large transformer-based backbones used in these models are bottlenecked by spatiotemporal attention. In this paper, we identify that a significant fraction of token-to-token connections consistently yield negligible scores across various inputs, and their patterns often repeat across queries. Thus, the attention computation in these cases can be skipped with little to no effect on the result. This observation continues to hold for connections among local token blocks. Motivated by this, we introduce CalibAtt, a training-free method that accelerates video generation via calibrated sparse attention. CalibAtt performs an offline calibration pass that identifies block-level sparsity and repetition patterns that are stable across inputs, and compiles these patterns into optimized attention operations for each layer, head, and diffusion timestep. At inference time, we compute the selected input-dependent connections densely, and skip the unselected ones in a hardware-efficient manner. Extensive experiments on Wan 2.1 14B, Mochi 1, and few-step distilled models at various resolutions show that CalibAtt achieves up to 1.58x end-to-end speedup, outperforming existing training-free methods while maintaining video generation quality and text-video alignment.

Problem

• text-to-video generation
• diffusion models
• spatiotemporal attention
• computational efficiency
• video generation speed

Innovation

• sparse attention
• video generation acceleration
• training-free optimization
• calibrated sparsity
• diffusion models