Dynamic 2D Gaussians: Geometrically accurate radiance fields for dynamic objects

2024-09-21

arXiv.org

AI Summary

Existing 4D implicit representations achieve high-quality novel-view synthesis but suffer from geometric inaccuracy, hindering high-fidelity dynamic mesh reconstruction from sparse views. To address this, we propose Dynamic 2D Gaussians (D-2DGS), an explicit geometric representation that models surfaces using differentiable, compact 2D Gaussian ellipsoids, driven spatiotemporally by sparse control points. We further introduce mask-guided depth clipping and rendering-mask-constrained isosurface extraction to enable end-to-end differentiable mesh reconstruction. To our knowledge, D-2DGS is the first explicit framework that jointly ensures geometric precision, efficient rendering, and direct mesh output. Quantitatively and qualitatively, it significantly outperforms state-of-the-art 4D implicit methods under sparse-view settings, enabling real-time rendering and high-fidelity temporally coherent mesh generation.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Data transparency and provenance
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reconstructing objects and extracting high-quality surfaces play a vital role in the real world. Current 4D representations show the ability to render high-quality novel views for dynamic objects but cannot reconstruct high-quality meshes due to their implicit or geometrically inaccurate representations. In this paper, we propose a novel representation that can reconstruct accurate meshes from sparse image input, named Dynamic 2D Gaussians (D-2DGS). We adopt 2D Gaussians for basic geometry representation and use sparse-controlled points to capture 2D Gaussian's deformation. By extracting the object mask from the rendered high-quality image and masking the rendered depth map, a high-quality dynamic mesh sequence of the object can be extracted. Experiments demonstrate that our D-2DGS is outstanding in reconstructing high-quality meshes from sparse input. More demos and code are available at https://github.com/hustvl/Dynamic-2DGS.

Problem

• Reconstructing high-quality dynamic object meshes from sparse images
• Overcoming implicit or geometrically inaccurate 4D representations
• Removing floaters to extract detailed and smooth mesh sequences

Innovation

• Dynamic 2D Gaussians for accurate mesh reconstruction
• Sparse-controlled points capture Gaussian deformation
• Mask-based floater removal for high-quality meshes