Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Dynamic Scenes

2024-04-18

AI Summary

To address structural detail loss and inter-frame geometric inconsistency in dynamic 3D reconstruction, this paper proposes a temporally consistent mesh reconstruction method based on 3D Gaussian Splatting. The method introduces three key innovations: (1) a Gaussian-mesh anchoring mechanism that explicitly couples Gaussian parameters with triangular mesh vertices; (2) a cycle-consistent deformation optimization framework, which jointly optimizes Gaussian distributions across frames via deformation-space mapping and canonical-space back-projection to enforce vertex trajectory consistency; and (3) a mesh-guided Gaussian densification and pruning strategy. Evaluated on multiple dynamic datasets, the approach achieves显著 improvements in mesh geometric accuracy and rendering fidelity, enhanced temporal coherence, and enables downstream applications such as dynamic texture editing.

Technology Category

• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Modern 3D engines and graphics pipelines require mesh as a memory-efficient representation, which allows efficient rendering, geometry processing, texture editing, and many other downstream operations. However, it is still highly difficult to obtain high-quality mesh in terms of detailed structure and time consistency from dynamic observations. To this end, we introduce Dynamic Gaussians Mesh (DG-Mesh), a framework to reconstruct a high-fidelity and time-consistent mesh from dynamic input. Our work leverages the recent advancement in 3D Gaussian Splatting to construct the mesh sequence with temporal consistency from dynamic observations. Building on top of this representation, DG-Mesh recovers high-quality meshes from the Gaussian points and can track the mesh vertices over time, which enables applications such as texture editing on dynamic objects. We introduce the Gaussian-Mesh Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh-guided densification and pruning on the deformed Gaussians. By applying cycle-consistent deformation between the canonical and the deformed space, we can project the anchored Gaussian back to the canonical space and optimize Gaussians across all time frames. During the evaluation on different datasets, DG-Mesh provides significantly better mesh reconstruction and rendering than baselines. Project page: https://www.liuisabella.com/DG-Mesh

Problem

• Reconstruct high-fidelity, time-consistent mesh from dynamic scenes.
• Enable efficient rendering and texture editing on dynamic objects.
• Improve mesh reconstruction using Gaussian-Mesh Anchoring technique.

Innovation

• Dynamic Gaussians Mesh reconstructs high-fidelity, time-consistent meshes.
• Uses 3D Gaussian Splatting for temporal consistency in dynamic scenes.
• Gaussian-Mesh Anchoring improves mesh quality via densification and pruning.