SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models

2026-06-26

AI Summary

This work addresses the "motion entanglement" problem in existing video diffusion models, where independent motion sources—such as camera and object movements—are erroneously coupled, leading to physically implausible videos. To mitigate this, the authors propose a Self-Imagined Fine-Tuning (SIFT) paradigm that eschews direct reconstruction of real videos and instead trains the model on its own generated samples. SIFT integrates motion-aware discriminative supervision with a progressive hard example replay strategy, leveraging free-form text prompts to encompass rare or finely disentangled motion scenarios and thereby alleviating data bias. Experimental results demonstrate that SIFT substantially enhances the physical plausibility, motion disentanglement, and controllability of generated videos, outperforming current methods across multiple quantitative metrics and visual evaluations.

Abstract

Recent advances in video diffusion models have greatly improved visual fidelity, yet their generated motions often violate physical plausibility. We observe a common kinematic failure, "motion entanglement", the unintended coupling of independent motion sources, such as camera movement and object motion. We identify that this issue stems from data bias and the reconstruction-based training design of diffusion models. Training on noisy videos that still retain coarse motion cues inadvertently encourages the model to replicate existing motion without an incentive to learn how to model kinematically-grounded motions. To address this, we propose a Self-Imagination Fine-Tuning (SIFT) paradigm, which enables the model to learn from its own generated videos rather than directly reconstructing real ones, breaking the reconstruction shortcut. We further employ motion-aware discriminative supervision and a progressive hard-case replay strategy to stabilize and accelerate learning. By leveraging freely-generated text prompts, our method can densely cover a broad motion space, including rare or finely-disentangled scenarios that would be costly to collect as video data. Extensive experiments demonstrate that our approach substantially improves the physical realism, motion disentanglement, and controllability of generated videos.

Problem

• physically plausible motion
• motion entanglement
• video diffusion models
• kinematic failure
• motion disentanglement

Innovation

• Self-Imagination Fine-Tuning
• motion disentanglement
• video diffusion models
• physical plausibility
• discriminative supervision