Go-with-the-Track: Video Compositing and Motion Control with Point Tracking

2026-06-18

AI Summary

Existing video generation methods struggle to simultaneously achieve precise synthesis from multiple reference images and fine-grained spatiotemporal motion control. This work proposes a unified framework that establishes explicit correspondences between generated frames and multiple references through spatially aware point trajectory embeddings, which serve as motion anchors to jointly condition the diffusion process. These embeddings are constructed using a coordinate-based MLP and temporal pooling, then injected into a video diffusion Transformer via lightweight adapters. Trained with a hybrid strategy combining static, dynamic, and synthetic data, the single model enables high-fidelity multi-reference video generation, point-trajectory-driven motion manipulation, and camera control, demonstrating superior reference consistency and motion controllability in both static and dynamic scenes.

Abstract

Filmmaking demands precise motion control and reference image compositing -- capabilities that existing methods treat separately. Point-track-conditioned image-to-video models restrict content insertion to the first frame, while reference-to-video models lack fine-grained spatial-temporal control over how reference content integrates across frames. We present Go-with-the-Track, which unifies both capabilities by jointly conditioning on multiple reference images and reference-anchored point-tracks -- extending conventional point-tracks to explicitly establish correspondences between generated frames and reference images, thus enabling precise compositing and motion control throughout the video. To achieve this, we introduce spatially-aware point-track embeddings that encode the full sequence of point-track coordinates using a coordinate-wise MLP followed by temporal pooling. This representation captures the spatial characteristics of each point-track (serving as a unique identifier), while the embedding similarity correlates directly with spatial proximity, enhancing the model's ability to distinguish and associate point-tracks. We inject these point-track embeddings into a video diffusion transformer via a lightweight adapter, resolving the pixel-to-patch resolution mismatch while avoiding the substantial motion detail loss inherent in naive point-track subsampling. We use a hybrid training strategy to train jointly on dynamic, static, and synthetic scene video datasets to boost motion controllability. Experiments demonstrate that Go-with-the-Track achieves superior motion and reference control in a single model and enables new capabilities: multi-reference conditioned video generation with point-track driven compositing, as well as camera control for both static and dynamic scenes. Project Page: https://eyeline-labs.github.io/Go-with-the-Track/

Problem

• video compositing
• motion control
• point tracking
• reference image
• spatial-temporal control

Innovation

• point tracking
• video compositing
• motion control
• diffusion transformer
• reference-conditioned generation