Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation

2024-12-08

arXiv.org

AI Summary

Object appearance drift in video generation—causing inter-frame visual discontinuity—remains a critical bottleneck for diffusion-based models. To address this, we propose an end-to-end joint optimization framework that, for the first time within the Stable Video Diffusion architecture, integrates differentiable point tracking directly into the diffusion training objective. Our method leverages inter-frame keypoint trajectories as explicit spatial supervision, jointly optimizing the standard video diffusion loss and a tracking loss enforcing optical flow consistency. Crucially, it requires no auxiliary modules, pretrained models, or manual annotations, unifying video generation and motion tracking within a single network. Experiments demonstrate substantial suppression of appearance drift, yielding significant improvements in motion coherence and temporal stability across multiple benchmarks. Notably, the Fréchet Video Distance (FVD) decreases by up to 32%, establishing a novel paradigm for spatiotemporal consistency modeling in video diffusion.

Technology Category

• Computer Vision: Motion & Tracking
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Tracking, profiling, and countermeasures against them

Abstract

While recent foundational video generators produce visually rich output, they still struggle with appearance drift, where objects gradually degrade or change inconsistently across frames, breaking visual coherence. We hypothesize that this is because there is no explicit supervision in terms of spatial tracking at the feature level. We propose Track4Gen, a spatially aware video generator that combines video diffusion loss with point tracking across frames, providing enhanced spatial supervision on the diffusion features. Track4Gen merges the video generation and point tracking tasks into a single network by making minimal changes to existing video generation architectures. Using Stable Video Diffusion as a backbone, Track4Gen demonstrates that it is possible to unify video generation and point tracking, which are typically handled as separate tasks. Our extensive evaluations show that Track4Gen effectively reduces appearance drift, resulting in temporally stable and visually coherent video generation. Project page: hyeonho99.github.io/track4gen

Problem

• Reducing appearance drift in video generation
• Combining video diffusion with point tracking
• Enhancing spatial supervision for visual coherence

Innovation

• Combines video diffusion with point tracking
• Unifies generation and tracking in one network
• Reduces appearance drift for stable videos