HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation

2026-07-22

AI Summary

This work addresses the high computational cost and temporal flickering in autoregressive video generation, caused by the growing key-value (KV) cache with frame count—especially at high resolutions. The authors propose HeadCast, a training-free, plug-and-play acceleration framework that reveals, for the first time, the stable heterogeneous behaviors of attention heads in pretrained models. Based on this insight, attention heads are categorized into four types: Sink, Dummy, Spatial, and Global. HeadCast leverages a one-time head classification, head-specific KV caching paths, and a fixed-grid Spatial pathway to drastically reduce computation while preserving long-range temporal consistency. Experiments show that HeadCast achieves up to 1.62× and 1.95× inference speedup at 720p and 1080p resolutions, respectively, maintaining VBench quality comparable to full attention with negligible flickering.

Abstract

Autoregressive (AR) video diffusion models have become a promising paradigm for long and streaming video synthesis, but the continuously growing Key-Value (KV) cache makes attention the dominant inference cost, especially at high resolution where each frame contributes many tokens. Existing remedies either evict the cache with coarse heuristics that cause inter-frame flickering, or require model re-training. We propose HeadCast, a training-free, plug-and-play acceleration framework built on the observation that a pre-trained AR model's attention heads exhibit stable, heterogeneous behaviors. After a short warm-up, HeadCast performs a one-time classification at the maximum-noise step that sorts every head into one of four archetypes: Sink, Dummy, Spatial, and Global, and restructures the monolithic KV cache into head-specific pathways. Crucially, it retains the Global heads that preserve the long-range temporal consistency aggressive eviction destroys. Because the Spatial pathway operates on a fixed-size grid, its savings grow with resolution: across state-of-the-art AR models, HeadCast accelerates inference by up to 1.62x at 720P and 1.95x at 1080P, while keeping VBench quality on par with full attention and largely flicker-free. Code is available at https://github.com/sjlgaga/HeadCast .

Problem

• autoregressive video generation
• attention mechanism
• KV cache
• inference efficiency
• temporal consistency

Innovation

• HeadCast
• attention head classification
• KV cache optimization
• autoregressive video generation
• training-free acceleration