Steering Video Diffusion Transformers with Massive Activations

2026-03-18

AI Summary

This work proposes Structured Activation Steering (STAS), a training-free, self-guided method that leverages the structured distribution of massive activations in video diffusion Transformers. By uncovering, for the first time, the hierarchical spatiotemporal patterns of massive activations within temporally chunked latent spaces, STAS selectively modulates activation values at critical positions to significantly enhance both visual quality and temporal coherence of generated videos with minimal computational overhead. Extensive experiments demonstrate consistent improvements across diverse text-to-video diffusion models, highlighting STAS’s efficiency, robustness, and broad applicability without requiring model retraining or architectural modifications.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Structured Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite rapid progress in video diffusion transformers, how their internal model signals can be leveraged with minimal overhead to enhance video generation quality remains underexplored. In this work, we study the role of Massive Activations (MAs), which are rare, high-magnitude hidden state spikes in video diffusion transformers. We observed that MAs emerge consistently across all visual tokens, with a clear magnitude hierarchy: first-frame tokens exhibit the largest MA magnitudes, latent-frame boundary tokens (the head and tail portions of each temporal chunk in the latent space) show elevated but slightly lower MA magnitudes than the first frame, and interior tokens within each latent frame remain elevated, yet are comparatively moderate in magnitude. This structured pattern suggests that the model implicitly prioritizes token positions aligned with the temporal chunking in the latent space. Based on this observation, we propose Structured Activation Steering (STAS), a training-free self-guidance-like method that steers MA values at first-frame and boundary tokens toward a scaled global maximum reference magnitude. STAS achieves consistent improvements in terms of video quality and temporal coherence across different text-to-video models, while introducing negligible computational overhead.

Problem

• Video Diffusion Transformers
• Massive Activations
• Video Generation Quality
• Temporal Coherence
• Model Internal Signals

Innovation

• Massive Activations
• Structured Activation Steering
• Video Diffusion Transformers
• Temporal Coherence
• Training-free Guidance