Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfer

2025-03-21

AI Summary

Video diffusion Transformers (DiTs) struggle to disentangle motion from appearance in motion transfer tasks, leading to inaccurate motion modeling. To address this, we propose DeT: a novel diffusion-based framework that (i) employs lightweight temporal convolutional kernels for spatiotemporal feature disentanglement; (ii) constructs dense trajectories in the latent space and enforces explicit motion consistency supervision; and (iii) integrates multi-scale motion similarity metrics with 3D feature distillation. Furthermore, we introduce MTBench—the first general-purpose motion transfer benchmark—and a hybrid motion fidelity metric. Extensive experiments demonstrate that DeT consistently outperforms state-of-the-art diffusion models in motion coherence, local dynamic detail preservation, and cross-scene generalization. Notably, it achieves a superior balance between motion fidelity and editing fidelity, advancing the practical applicability of diffusion models for high-fidelity motion transfer.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Deep Generative Models & Autoencoders
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The motion transfer task involves transferring motion from a source video to newly generated videos, requiring the model to decouple motion from appearance. Previous diffusion-based methods primarily rely on separate spatial and temporal attention mechanisms within 3D U-Net. In contrast, state-of-the-art video Diffusion Transformers (DiT) models use 3D full attention, which does not explicitly separate temporal and spatial information. Thus, the interaction between spatial and temporal dimensions makes decoupling motion and appearance more challenging for DiT models. In this paper, we propose DeT, a method that adapts DiT models to improve motion transfer ability. Our approach introduces a simple yet effective temporal kernel to smooth DiT features along the temporal dimension, facilitating the decoupling of foreground motion from background appearance. Meanwhile, the temporal kernel effectively captures temporal variations in DiT features, which are closely related to motion. Moreover, we introduce explicit supervision along dense trajectories in the latent feature space to further enhance motion consistency. Additionally, we present MTBench, a general and challenging benchmark for motion transfer. We also introduce a hybrid motion fidelity metric that considers both the global and local motion similarity. Therefore, our work provides a more comprehensive evaluation than previous works. Extensive experiments on MTBench demonstrate that DeT achieves the best trade-off between motion fidelity and edit fidelity.

Problem

• Improve motion transfer in video Diffusion Transformers (DiT)
• Decouple motion and appearance via temporal kernel smoothing
• Enhance motion consistency with dense trajectory supervision

Innovation

• Introduces temporal kernel for motion decoupling
• Uses dense trajectory supervision for consistency
• Proposes hybrid metric for motion evaluation