IM360: Textured Mesh Reconstruction for Large-scale Indoor Mapping with 360$^circ$ Cameras

2025-02-18

AI Summary

提出IM360方法，利用360度相机和球形相机模型改进SfM流程，结合神经隐式表面重建和基于网格的神经渲染技术，解决大规模室内场景三维重建中的纹理缺失和重复区域问题。

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We present a novel 3D reconstruction pipeline for 360$^circ$ cameras for 3D mapping and rendering of indoor environments. Traditional Structure-from-Motion (SfM) methods may not work well in large-scale indoor scenes due to the prevalence of textureless and repetitive regions. To overcome these challenges, our approach (IM360) leverages the wide field of view of omnidirectional images and integrates the spherical camera model into every core component of the SfM pipeline. In order to develop a comprehensive 3D reconstruction solution, we integrate a neural implicit surface reconstruction technique to generate high-quality surfaces from sparse input data. Additionally, we utilize a mesh-based neural rendering approach to refine texture maps and accurately capture view-dependent properties by combining diffuse and specular components. We evaluate our pipeline on large-scale indoor scenes from the Matterport3D and Stanford2D3D datasets. In practice, IM360 demonstrate superior performance in terms of textured mesh reconstruction over SOTA. We observe accuracy improvements in terms of camera localization and registration as well as rendering high frequency details.

Problem

• Reconstructs 3D indoor environments using 360° cameras
• Improves textureless and repetitive region handling
• Enhances mesh reconstruction and rendering accuracy

Innovation

• 360° camera 3D mapping
• neural implicit surface reconstruction
• mesh-based neural rendering