Real-Time Video Generation with Pyramid Attention Broadcast

2024-08-22

arXiv.org

AI Summary

Diffusion-based video generation using DiT architectures suffers from high computational redundancy in attention computation, slow inference, and reliance on pretraining. Method: This paper proposes Pyramid Attention Broadcast (PAB), a training-free method that identifies a U-shaped redundancy pattern in attention outputs across diffusion timesteps and designs a pyramid-strided broadcasting mechanism to exploit it. It further introduces hierarchical variance-adaptive scheduling and sequence-level parallel inference to minimize redundant computations. Contribution/Results: PAB achieves 10.5× speedup across three DiT variants without any training overhead, enabling real-time 720p video generation. It preserves generation quality while significantly improving inference efficiency—addressing key bottlenecks in diffusion-based video synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

We present Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free approach for DiT-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style. It applies different broadcast strategies to each attention based on their variance for best efficiency. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates up to 10.5x speedup across three models compared to baselines, achieving real-time generation for up to 720p videos. We anticipate that our simple yet effective method will serve as a robust baseline and facilitate future research and application for video generation.

Problem

• Video Generation
• Efficiency Improvement
• Attention Mechanism Optimization

Innovation

• Pyramid Attention Broadcasting
• Video Generation Acceleration
• Real-time 720p Video