Dynamic-in-Few-Step: Unifying Dynamic Computation and Few-Step Distillation for Efficient Video Generation

2026-07-07

AI Summary

Video diffusion models suffer from high inference costs, and existing few-step distillation methods employ static architectures that fail to adapt to the varying computational demands across denoising stages. This work proposes a post-training acceleration framework that jointly optimizes dynamic structural sparsity and the number of denoising steps during distillation, for the first time enabling step-specific Mixture-of-Models (MoM). A progressive training strategy combined with an output unfolding mechanism stabilizes the optimization process, complemented by an efficient inference engine. Evaluated on the Wan-14B model, the method reduces per-step FLOPs by 24% over a 4-step distilled baseline, achieving a 1.2× measured speedup and a 30× acceleration relative to the 50-step teacher model, all while preserving high-quality generation.

Abstract

Video Diffusion Models (VDMs) have demonstrated superior generation quality but suffer from prohibitive computational costs. While recent few-step distillation techniques significantly accelerate inference, they typically enforce a static model architecture across all denoising stages, ignoring the varying computational demands inherent to different noise levels. In this work, we propose a novel post-training acceleration framework that exploits this redundancy by integrating dynamic structural sparsification directly into the distillation process. Unlike conventional post-hoc compression applied to a fixed diffusion pipeline, our approach jointly optimizes the denoising steps and structured model sparsity, transforming a pre-trained VDM into a compact, step-specific Mixture-of-Models (MoM). To address the training instability arising from this joint optimization, we introduce a Progressive Training Strategy coupled with an Output Rollout Mechanism, which ensures the coherent learning of structural decisions across timesteps. Furthermore, we develop a specialized inference engine to deploy the resulting MoM efficiently. Our method is orthogonal to existing acceleration techniques and highly effective: On Wan-14B, it removes 24% of the per-step FLOPs on top of 4-step distillation, adding a 1.2x wall-clock gain and reaching a 30x speedup over the 50-step teacher while preserving competitive generation quality.

Problem

• Video Diffusion Models
• Few-Step Distillation
• Dynamic Computation
• Computational Efficiency
• Model Sparsity

Innovation

• dynamic computation
• few-step distillation
• structured sparsity
• Mixture-of-Models
• video diffusion models