MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation

2026-06-24

AI Summary

This work addresses the challenge of simultaneously preserving geometric consistency and motion fidelity in novel view synthesis from monocular video. To this end, the authors propose a motion-aware training framework that leverages multi-view point tracking to provide joint geometric and motion supervision, explicitly reinforcing correspondences across views and over time. A key insight is that certain attention layers in diffusion models inherently encode strong correspondence cues; building on this observation, the method introduces an auxiliary multi-view tracking head trained jointly with the main generation pipeline to align spatio-temporal features across views under camera conditioning. Experiments demonstrate that the proposed approach significantly improves geometric consistency on multiple benchmarks while maintaining state-of-the-art accuracy in camera trajectory estimation.

Abstract

Synthesizing a novel-view video from a monocular reference video along a target camera trajectory requires both geometric consistency and motion fidelity with respect to the reference video. Existing methods based on explicit 3D representations are limited by the accuracy of off-the-shelf reconstruction modules, which often produce inaccurate geometry for dynamic objects in monocular videos. In contrast, camera-conditioning-only methods can achieve high visual quality but often struggle to preserve geometric and motion consistency. In this work, we introduce MVTrack4Gen (Multi-View point Tracking for Novel-View Generation), a motion-aware training framework that leverages multi-view point tracking as an additional geometric and motion supervision signal for camera-conditioning-only novel-view video diffusion models. Our key finding is that specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations across views and over time, and the misalignment of these correspondences causes motion inconsistency. Based on this observation, we route these features into an auxiliary multi-view tracking head and jointly train the diffusion model with a point-tracking objective. By explicitly strengthening these motion-aware correspondences, MVTrack4Gen improves existing models to better follow the motion in the reference view and maintain cross-view geometric consistency. Across diverse benchmarks, our method achieves state-of-the-art geometric consistency and competitive camera accuracy.

Problem

• novel-view video generation
• geometric consistency
• motion fidelity
• monocular video
• 4D video synthesis

Innovation

• multi-view point tracking
• geometric supervision
• 4D video generation
• motion consistency
• diffusion models