TrajVG: 3D Trajectory-Coupled Visual Geometry Learning

2026-02-04

AI Summary

This work addresses the challenges of drift, misalignment, and structural duplication in multi-frame feedforward 3D reconstruction of dynamic scenes, which arise from ambiguous global references and over-reliance on local point maps. To overcome these issues, the authors propose explicitly predicting sparse 3D trajectories in camera coordinates and jointly optimizing them with per-frame local point maps and relative camera poses. The method introduces two key innovations: a bidirectional trajectory–point map consistency constraint and a pose consistency loss based on static anchor points. Notably, it is the first to treat 3D trajectories as an explicit supervision target, enabling unified training under mixed supervision. The approach significantly outperforms existing feedforward methods across multiple tasks, including 3D tracking, camera pose estimation, point map reconstruction, and video depth estimation.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Feed-forward multi-frame 3D reconstruction models often degrade on videos with object motion. Global-reference becomes ambiguous under multiple motions, while the local pointmap relies heavily on estimated relative poses and can drift, causing cross-frame misalignment and duplicated structures. We propose TrajVG, a reconstruction framework that makes cross-frame 3D correspondence an explicit prediction by estimating camera-coordinate 3D trajectories. We couple sparse trajectories, per-frame local point maps, and relative camera poses with geometric consistency objectives: (i) bidirectional trajectory-pointmap consistency with controlled gradient flow, and (ii) a pose consistency objective driven by static track anchors that suppresses gradients from dynamic regions. To scale training to in-the-wild videos where 3D trajectory labels are scarce, we reformulate the same coupling constraints into self-supervised objectives using only pseudo 2D tracks, enabling unified training with mixed supervision. Extensive experiments across 3D tracking, pose estimation, pointmap reconstruction, and video depth show that TrajVG surpasses the current feedforward performance baseline.

Problem

• 3D reconstruction
• object motion
• cross-frame misalignment
• trajectory estimation
• visual geometry

Innovation

• 3D trajectory
• visual geometry learning
• self-supervised reconstruction
• geometric consistency
• multi-frame 3D reconstruction