Data-Forcing Distillation: Restoring Diversity and Fidelity in Few-Step Video Generation

2026-06-16

AI Summary

Existing few-step video generation distillation methods often suffer from reduced sample diversity and output oversaturation due to their reliance on the reverse KL divergence objective, leading to deviations from the true data distribution. This work proposes Data-Forcing Distillation (DFD), a post-distillation framework requiring only a single-line code modification, which leverages score differences from the teacher model to guide the student toward the authentic data distribution. DFD effectively mitigates mode collapse and eliminates spurious modes, uniquely restoring both generation diversity and fidelity in the post-distillation phase with theoretical clarity and implementation simplicity. Experiments on Wan2.1-1.3B and Cosmos-Predict2.5-2B demonstrate that merely 100–300 fine-tuning steps significantly enhance video dynamics and visual quality, remove oversaturation artifacts, and even surpass the teacher model’s performance.

Abstract

Recent progress has shown promise in distilling multi-step video diffusion models into efficient few-step students. Among them, Distribution Matching Distillation (DMD) and its successor DMD2 achieved strong generation quality and fast convergence. However, due to the nature of the reverse Kullback--Leibler (KL) objective, these methods exhibit two persistent failure modes: a substantial drop in sample diversity, and visibly over-saturated outputs that deviate from real-video appearance. In this work, we propose Data-Forcing Distillation (DFD), a simple post-training framework that restores diversity and fidelity in DMD with only a single-line of code change. At its core is the teacher score discrepancy to guide the student toward the real-data distribution, pulling it to missing modes (mitigating mode collapse) and away from problematic modes absent in real data (avoiding over-saturation). We provide an in-depth theoretical analysis of our framework and validate our approach on text-to-video, image-to-video, and autoregressive video generation. With only 100--300 steps of finetuning, DFD effectively restores diversity and fidelity on both Wan2.1-1.3B and Cosmos-Predict2.5-2B model, resolving the over-saturation artifacts with significantly better video dynamics and appearance, and even outperforms the teacher model.

Problem

• few-step video generation
• sample diversity
• over-saturation
• distribution fidelity
• distillation

Innovation

• Data-Forcing Distillation
• video diffusion models
• few-step generation
• mode collapse
• distribution alignment