GSTAR: Gaussian Surface Tracking and Reconstruction

2025-01-17

AI Summary

Existing 3D Gaussian-based methods struggle with topological changes in dynamic objects—such as appearance, disappearance, and splitting—leading to tracking drift and geometric distortion. To address this, we propose a Gaussian-mesh co-modeling framework featuring a novel bindable/unbindable Gaussian-mesh coupling mechanism. Our method integrates surface-flow-driven inter-frame initialization and adaptive Gaussian decoupling to jointly model topological evolution and geometric optimization of dynamic surfaces for the first time. It unifies 3D Gaussian splatting, deformable mesh regularization, surface-based scene flow estimation, and multi-view joint optimization. Evaluated on multiple topologically varying dynamic scenes, our approach achieves state-of-the-art tracking accuracy and high-fidelity surface reconstruction, while enabling real-time rendering and interactive editing.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Learning with Manifolds
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Web Mining and Content Analysis: Models for Web evolution

Abstract

3D Gaussian Splatting techniques have enabled efficient photo-realistic rendering of static scenes. Recent works have extended these approaches to support surface reconstruction and tracking. However, tracking dynamic surfaces with 3D Gaussians remains challenging due to complex topology changes, such as surfaces appearing, disappearing, or splitting. To address these challenges, we propose GSTAR, a novel method that achieves photo-realistic rendering, accurate surface reconstruction, and reliable 3D tracking for general dynamic scenes with changing topology. Given multi-view captures as input, GSTAR binds Gaussians to mesh faces to represent dynamic objects. For surfaces with consistent topology, GSTAR maintains the mesh topology and tracks the meshes using Gaussians. In regions where topology changes, GSTAR adaptively unbinds Gaussians from the mesh, enabling accurate registration and the generation of new surfaces based on these optimized Gaussians. Additionally, we introduce a surface-based scene flow method that provides robust initialization for tracking between frames. Experiments demonstrate that our method effectively tracks and reconstructs dynamic surfaces, enabling a range of applications. Our project page with the code release is available at https://chengwei-zheng.github.io/GSTAR/.

Problem

• 3D Gaussian Techniques
• Dynamic Shape Variability
• Object Appearance and Disappearance

Innovation

• GSTAR
• Gaussian patterns
• scene flow