G$^2$TAM: Geometry Grounded Track Anything Model

2026-07-04

AI Summary

This work addresses the challenge that existing video segmentation methods, which rely on explicit appearance-based memory, struggle to maintain spatial consistency across viewpoints and over time under large viewpoint changes and prolonged occlusions. To overcome this limitation, the authors propose a unified framework that, for the first time, leverages spatially aligned geometric representations from a feed-forward 3D reconstruction model as implicit memory for promptable 3D instance tracking. By introducing a cross-modal spatial encoder that fuses visual and textual prompts into a shared geometric space, the framework enables end-to-end spatial reconstruction and consistent mask prediction. Experiments on the newly introduced large-scale dataset InsTrack demonstrate state-of-the-art performance in cross-view consistency, promptable tracking, video object segmentation, and 3D reconstruction.

Abstract

Human spatial understanding arises from jointly perceiving geometry and semantics, enabling consistent object identification and localization across viewpoints and time. Current video segmentation models depend on explicit object appearance memory banks for instance tracking, yet they remain vulnerable to large viewpoint changes and long-term occlusions. Leveraging the spatial consistency afforded by modern feed-forward 3D reconstruction models, we propose the Geometry Grounded Tracking Anything Model (G$^2$TAM), a unified framework for promptable instance tracking in 3D using only unordered RGB images or videos. G$^2$TAM employs spatially aligned geometric representations as implicit memory, ensuring stable instance identity and localization across frames and views. At its core is a cross-modal spatial encoder that integrates visual and textual prompts into a shared geometric space, enabling end-to-end spatial reconstruction and instance-consistent mask prediction. To support training and evaluation, we construct InsTrack, a large-scale dataset with a dedicated validation split for benchmarking. Extensive experiments show that G$^2$TAM delivers strong cross-view consistency, promptable instance spatial tracking, video object segmentation and spatial reconstruction, establishing a foundation for interactive, geometry-grounded spatial reasoning.

Problem

• video segmentation
• instance tracking
• viewpoint changes
• occlusions
• spatial consistency

Innovation

• geometry-grounded tracking
• 3D instance segmentation
• spatially aligned representation
• cross-modal spatial encoder
• implicit memory