HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models

2026-08-13

AI Summary

This work aims to effectively transfer the superior generation capabilities of Text-Image-to-Video (TI2V) models—exhibited when conditioned on high-quality initial frames or detailed textual prompts—to their underlying Text-to-Video (T2V) task. To this end, the authors propose a hybrid self-distillation framework in which a single model acts as both teacher (in TI2V mode) and student (in T2V mode). The approach leverages off-policy trajectory anchoring, local policy optimization, and velocity-level supervision signals to enable precise policy correction while avoiding condition-state mismatches. Notably, this method is the first to integrate privileged priors with online policy fine-tuning, significantly enhancing base T2V generation quality without requiring additional data, while also further improving TI2V performance—thereby comprehensively strengthening the model’s video synthesis capabilities.

Abstract

Text-Image-to-Video (TI2V) models are an emerging unified architecture, where a single model simultaneously supports text-to-video (T2V) and image-to-video (I2V) generation. Given a high-quality first frame or a detailed textual prompt, TI2V models unlock substantially better visual quality than their T2V mode, raising a natural question: can the capability elicited by such privileged conditions be internalized into the model's own base generation ability? A common approach toward this goal is model self-distillation. However, the most straightforward solution, supervised fine-tuning, follows an off-policy strategy: its supervision is confined to teacher-generated endpoints from a fixed offline distribution rather than student-visited states, lacking precise correction tailored to the evolving policy. Recent on-policy distillation methods instead suffer from condition-state mismatch, where supervision is steered toward the given first frame instead of the student's actual content, misleading the correction. To achieve self-distillation that absorbs the teacher's privileged prior while retaining precise policy correction, in this work, we propose Hybrid-Policy Self-Distillation (HPSD), a novel self-distillation framework where a single TI2V model acts as both teacher and student under different conditions: the teacher operates in TI2V mode with a high-quality first frame and an enhanced prompt, while the student runs in the base T2V mode with only the vanilla prompt. Specifically, the student inherits off-policy teacher trajectory points as anchors, locally refines them toward its own policy, and finally receives velocity-level supervision on these self-generated roll-outs. Extensive experiments demonstrate that HPSD significantly improves T2V performance while also delivering notable TI2V gains, effectively strengthening the model's base generation ability.

Problem

• Text-Image-to-Video
• self-distillation
• text-to-video
• policy correction
• generation ability

Innovation

• Hybrid-Policy Self-Distillation
• Text-Image-to-Video
• Diffusion Models
• Self-Distillation
• Policy Correction