Head Forcing: Long Autoregressive Video Generation via Head Heterogeneity

2026-05-14

AI Summary

Autoregressive video diffusion models are prone to error accumulation and contextual degradation during long-duration generation. This work reveals, for the first time, functional heterogeneity among attention heads in their underlying Transformers and introduces Head Forcing—a training-free framework that leverages this insight. By designing head-specific KV caching strategies for local, anchor, and memory heads, integrating head-level RoPE positional re-encoding, and incorporating a dynamic context updating mechanism, the method constructs a hierarchical memory system. This approach substantially enhances generation stability and temporal consistency, extending video synthesis from 5 seconds to minute-scale durations while enabling multi-prompt interactive editing. The proposed method outperforms existing baselines across multiple evaluation metrics.

Abstract

Autoregressive video diffusion models support real-time synthesis but suffer from error accumulation and context loss over long horizons. We discover that attention heads in AR video diffusion transformers serve functionally distinct roles as local heads for detail refinement, anchor heads for structural stabilization, and memory heads for long-range context aggregation, yet existing methods treat them uniformly, leading to suboptimal KV cache allocation. We propose Head Forcing, a training-free framework that assigns each head type a tailored KV cache strategy: local and anchor heads retain only essential tokens, while memory heads employ a hierarchical memory system with dynamic episodic updates for long-range consistency. A head-wise RoPE re-encoding scheme further ensures positional encodings remain within the pretrained range. Without additional training, Head Forcing extends generation from 5 seconds to minute-level duration, supports multi-prompt interactive synthesis, and consistently outperforms existing baselines. Project Page: https://jiahaotian-sjtu.github.io/headforcing.github.io/.

Problem

• autoregressive video generation
• error accumulation
• context loss
• long-horizon generation
• KV cache allocation

Innovation

• Head Forcing
• attention head heterogeneity
• KV cache optimization
• long autoregressive video generation
• RoPE re-encoding