OmniIndoor3D: Comprehensive Indoor 3D Reconstruction

2025-05-27

AI Summary

Existing 3D Gaussian Splatting (3DGS)-based indoor reconstruction methods prioritize photorealistic rendering at the expense of geometric accuracy, limiting their utility for panoptic understanding and robotic navigation. To address this, we propose the first end-to-end appearance-geometry-panoptic reconstruction framework tailored for RGB-D input. Our method comprises three key innovations: (1) RGB-D–guided coarse reconstruction for robust 3DGS initialization; (2) a lightweight MLP architecture that decouples and jointly optimizes appearance and geometry parameters; and (3) panoptic segmentation priors to drive adaptive Gaussian densification, significantly improving planarity continuity and geometric fidelity. Evaluated on multiple indoor benchmarks, our approach achieves state-of-the-art performance across rendering quality (PSNR, SSIM, LPIPS), geometric accuracy (Chamfer distance, F-Score), and panoptic segmentation metrics (PQ, SQ, RQ), effectively bridging the long-standing gap between visual realism and geometric reliability in 3D reconstruction.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We propose a novel framework for comprehensive indoor 3D reconstruction using Gaussian representations, called OmniIndoor3D. This framework enables accurate appearance, geometry, and panoptic reconstruction of diverse indoor scenes captured by a consumer-level RGB-D camera. Since 3DGS is primarily optimized for photorealistic rendering, it lacks the precise geometry critical for high-quality panoptic reconstruction. Therefore, OmniIndoor3D first combines multiple RGB-D images to create a coarse 3D reconstruction, which is then used to initialize the 3D Gaussians and guide the 3DGS training. To decouple the optimization conflict between appearance and geometry, we introduce a lightweight MLP that adjusts the geometric properties of 3D Gaussians. The introduced lightweight MLP serves as a low-pass filter for geometry reconstruction and significantly reduces noise in indoor scenes. To improve the distribution of Gaussian primitives, we propose a densification strategy guided by panoptic priors to encourage smoothness on planar surfaces. Through the joint optimization of appearance, geometry, and panoptic reconstruction, OmniIndoor3D provides comprehensive 3D indoor scene understanding, which facilitates accurate and robust robotic navigation. We perform thorough evaluations across multiple datasets, and OmniIndoor3D achieves state-of-the-art results in appearance, geometry, and panoptic reconstruction. We believe our work bridges a critical gap in indoor 3D reconstruction. The code will be released at: https://ucwxb.github.io/OmniIndoor3D/

Problem

• Accurate indoor 3D reconstruction using Gaussian representations
• Decoupling appearance and geometry optimization conflicts
• Improving Gaussian primitive distribution for panoptic reconstruction

Innovation

• Combines RGB-D images for coarse 3D initialization
• Uses lightweight MLP to adjust Gaussian geometry
• Proposes panoptic-guided densification for smooth surfaces