DUET: A Diversity-Quality Duet of Distillation Experts for Two-Step Video Generation

2026-08-10

AI Summary

Existing two-step video generation methods face a trade-off between quality and diversity: trajectory-level distillation approaches like sCM prioritize diversity at the expense of generation quality, whereas distribution-level distillation methods such as DMD ensure high fidelity but sacrifice structural variety. This work proposes DUET, a novel framework that introduces a dual-expert collaboration mechanism at the noise level—leveraging the sCM expert to generate diverse structures in high-noise stages and the DMD expert to refine fine details in low-noise stages. Guided by reinforcement learning–based expert adaptation, DUET overcomes bottlenecks at the relay interface and high-noise regime without resorting to complex loss-function combinations. Evaluated on the Wan2.1-T2V-1.3B backbone, DUET brings sCM’s generation quality close to that of DMD while preserving approximately twice the structural diversity; its enhanced variant, DUET+, further improves overall quality without compromising this diversity advantage.

Abstract

Diffusion models have enabled high-quality video generation in recent years, but the high cost of iterative sampling hinders their practical deployment. Few-step distillation alleviates this cost, yet exposes a quality--diversity trade-off between its two dominant paradigms: trajectory-level distillation (e.g., sCM) favors diversity, whereas distribution-level distillation (e.g., DMD) favors quality. Targeting extreme two-step video generation, we introduce DUET, which reconciles the two paradigms through a noise-level duet of experts: an sCM expert takes the high-noise step to lay out diverse structure, and a DMD expert takes the low-noise step to refine appearance detail. Since the two experts are trained independently with their native objectives, DUET sidesteps the optimization difficulties of loss-level combinations and delivers quality and diversity jointly rather than trading one for the other. We further identify the relay interface and the high-noise stage as the remaining bottlenecks, and address them with RL-guided expert adaptation, yielding DUET+. With the Wan2.1-T2V-1.3B backbone, DUET lifts the two-step quality of sCM close to the level of DMD while retaining nearly all of its structural diversity---about twice that of DMD---and DUET+ further improves overall quality while preserving this diversity advantage. Together, these results establish noise-level expert specialization as a simple, effective paradigm for reconciling diversity and quality in two-step video generation.

Problem

• video generation
• distillation
• quality-diversity trade-off
• two-step generation
• diffusion models

Innovation

• two-step video generation
• distillation experts
• quality-diversity trade-off
• noise-level specialization
• RL-guided adaptation