GTR: Gaussian Splatting Tracking and Reconstruction of Unknown Objects Based on Appearance and Geometric Complexity

2025-05-17

AI Summary

This work addresses the challenging problem of 6-degree-of-freedom (6-DoF) pose tracking and high-fidelity 3D reconstruction for symmetric, geometrically complex, or appearance-varying objects in monocular RGB-D video. We propose the first Gaussian Splatting-based tracking framework that jointly optimizes geometry and appearance while adaptively modeling object complexity—both geometric and photometric—through a novel appearance- and geometry-aware adaptive modeling scheme. To support open-world deployment, we design an adaptive keyframe selection strategy that dynamically balances tracking robustness and map fidelity. Furthermore, we introduce a new RGB-D tracking and reconstruction benchmark for objective evaluation. Extensive experiments demonstrate that our method significantly improves mesh reconstruction accuracy and 6-DoF tracking stability on complex objects, consistently outperforming state-of-the-art approaches. To our knowledge, this is the first method achieving robust, high-fidelity 3D perception and reconstruction from a single RGB-D sensor under open-world conditions.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Security and Privacy: Tracking, profiling, and countermeasures against them
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present a novel method for 6-DoF object tracking and high-quality 3D reconstruction from monocular RGBD video. Existing methods, while achieving impressive results, often struggle with complex objects, particularly those exhibiting symmetry, intricate geometry or complex appearance. To bridge these gaps, we introduce an adaptive method that combines 3D Gaussian Splatting, hybrid geometry/appearance tracking, and key frame selection to achieve robust tracking and accurate reconstructions across a diverse range of objects. Additionally, we present a benchmark covering these challenging object classes, providing high-quality annotations for evaluating both tracking and reconstruction performance. Our approach demonstrates strong capabilities in recovering high-fidelity object meshes, setting a new standard for single-sensor 3D reconstruction in open-world environments.

Problem

• 6-DoF object tracking from monocular RGBD video
• High-quality 3D reconstruction of complex objects
• Handling symmetry, intricate geometry, and complex appearance

Innovation

• Uses 3D Gaussian Splatting for reconstruction
• Combines hybrid geometry/appearance tracking
• Adaptive key frame selection for robustness