NeuManifold: Neural Watertight Manifold Reconstruction with Efficient and High-Quality Rendering Support

2023-05-26

arXiv.org

AI Summary

This work addresses the challenge of reconstructing high-fidelity, watertight manifold 3D meshes from multi-view images. We propose an implicit-explicit co-optimization framework: a neural voxel field provides robust geometric initialization, while differentiable rasterization enables topology-accurate optimization of watertight meshes, jointly learning a compact neural texture. Our approach uniquely unifies the robustness of neural implicit initialization with the topological fidelity of rasterization-based rendering, balancing reconstruction accuracy and real-time rendering capability. Quantitatively, our method achieves reconstruction quality comparable to NeRF-based volumetric rendering methods, while accelerating rendering by up to 10×. Crucially, it outputs standard triangle meshes—enabling texture mapping and direct integration into downstream 3D applications such as physics simulation and AR/VR.

Abstract

We present a method for generating high-quality watertight manifold meshes from multi-view input images. Existing volumetric rendering methods are robust in optimization but tend to generate noisy meshes with poor topology. Differentiable rasterization-based methods can generate high-quality meshes but are sensitive to initialization. Our method combines the benefits of both worlds; we take the geometry initialization obtained from neural volumetric fields, and further optimize the geometry as well as a compact neural texture representation with differentiable rasterizers. Through extensive experiments, we demonstrate that our method can generate accurate mesh reconstructions with faithful appearance that are comparable to previous volume rendering methods while being an order of magnitude faster in rendering. We also show that our generated mesh and neural texture reconstruction is compatible with existing graphics pipelines and enables downstream 3D applications such as simulation. Project page: https://sarahweiii.github.io/neumanifold/

Problem

• 3D Model Generation
• Image-based Modeling
• High-quality Detail Preservation

Innovation

• NeuManifold
• 3D model generation
• neural networks