TrackGo: A Flexible and Efficient Method for Controllable Video Generation

2024-08-21

arXiv.org

AI Summary

Existing video generation methods suffer from significant limitations in fine-grained content control, complex motion modeling, and background co-evolution. To address these challenges, we propose a novel framework for precise and controllable video generation. Our approach introduces a joint conditioning mechanism combining free-form masks and vector arrows—enabling part-level editing and explicit motion trajectory guidance. We further design TrackAdapter, a lightweight module embedded within temporal self-attention layers, to dynamically activate motion-relevant regions and enforce inter-frame consistency. Built upon diffusion models, our method is plug-and-play without requiring additional training. Quantitatively, it achieves state-of-the-art performance across standard metrics—including FVD, FID, and ObjMC—demonstrating substantial improvements in motion controllability, structural fidelity, and temporal coherence. This work establishes a new paradigm for controllable video generation in complex scenes.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent years have seen substantial progress in diffusion-based controllable video generation. However, achieving precise control in complex scenarios, including fine-grained object parts, sophisticated motion trajectories, and coherent background movement, remains a challenge. In this paper, we introduce TrackGo, a novel approach that leverages free-form masks and arrows for conditional video generation. This method offers users with a flexible and precise mechanism for manipulating video content. We also propose the TrackAdapter for control implementation, an efficient and lightweight adapter designed to be seamlessly integrated into the temporal self-attention layers of a pretrained video generation model. This design leverages our observation that the attention map of these layers can accurately activate regions corresponding to motion in videos. Our experimental results demonstrate that our new approach, enhanced by the TrackAdapter, achieves state-of-the-art performance on key metrics such as FVD, FID, and ObjMC scores.

Problem

• Video Generation
• Content Detail Control
• Complex Motion Handling

Innovation

• TrackGo
• TrackAdapter
• Video Generation Quality