MeshLRM: Large Reconstruction Model for High-Quality Mesh

2024-04-18

arXiv.org

AI Summary

Sparse-view 3D reconstruction suffers from heavy reliance on numerous input images, high computational cost, and complex multi-stage pipelines. Method: This paper introduces the first mesh-guided Large Reconstruction Model (LRM) framework. It pioneers the integration of differentiable mesh extraction and rendering into LRM, enabling end-to-end 3D mesh generation. The LRM architecture is simplified, and a sequential low- to high-resolution NeRF initialization strategy is proposed to significantly accelerate convergence and improve geometric fidelity. Contribution/Results: Given only four input views, our method generates high-fidelity watertight meshes in under one second—achieving state-of-the-art performance in sparse-view reconstruction. Moreover, it generalizes effectively to downstream tasks including text-to-3D and single-image-to-3D synthesis. To our knowledge, this is the first approach to jointly achieve both high reconstruction quality and ultra-fast inference (<1 s), marking a breakthrough in efficiency–fidelity trade-off for 3D generative modeling.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We propose MeshLRM, a novel LRM-based approach that can reconstruct a high-quality mesh from merely four input images in less than one second. Different from previous large reconstruction models (LRMs) that focus on NeRF-based reconstruction, MeshLRM incorporates differentiable mesh extraction and rendering within the LRM framework. This allows for end-to-end mesh reconstruction by fine-tuning a pre-trained NeRF LRM with mesh rendering. Moreover, we improve the LRM architecture by simplifying several complex designs in previous LRMs. MeshLRM's NeRF initialization is sequentially trained with low- and high-resolution images; this new LRM training strategy enables significantly faster convergence and thereby leads to better quality with less compute. Our approach achieves state-of-the-art mesh reconstruction from sparse-view inputs and also allows for many downstream applications, including text-to-3D and single-image-to-3D generation. Project page: https://sarahweiii.github.io/meshlrm/

Problem

• 3D Model Generation
• NeRF-based Methods
• Resource Efficiency

Innovation

• MeshLRM
• Low-to-High Resolution Training
• 3D Mesh Generation