NexusSplats: Efficient 3D Gaussian Splatting in the Wild

2024-11-21

arXiv.org

AI Summary

To address low-fidelity 3D reconstruction under complex real-world illumination variations and transient occlusions, this paper proposes a structure-aware reconstruction method based on 3D Gaussian Splatting (3DGS). Our approach tackles two key challenges: (1) a hierarchical illumination disentanglement strategy that enables centralized appearance modeling and improves illumination generalization; and (2) a 3D–2D structural correspondence-driven occlusion reasoning mechanism supporting structure-agnostic dynamic occlusion handling. By jointly optimizing hierarchical illumination representations and structure-guided occlusion masks, our method achieves state-of-the-art rendering quality while reducing model parameters by 65.4% and accelerating reconstruction speed by 2.7×. The proposed framework significantly enhances robustness and efficiency in uncontrolled, complex outdoor environments.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Photorealistic 3D reconstruction of unstructured real-world scenes remains challenging due to complex illumination variations and transient occlusions. Existing methods based on Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) struggle with inefficient light decoupling and structure-agnostic occlusion handling. To address these limitations, we propose NexusSplats, an approach tailored for efficient and high-fidelity 3D scene reconstruction under complex lighting and occlusion conditions. In particular, NexusSplats leverages a hierarchical light decoupling strategy that performs centralized appearance learning, efficiently and effectively decoupling varying lighting conditions. Furthermore, a structure-aware occlusion handling mechanism is developed, establishing a nexus between 3D and 2D structures for fine-grained occlusion handling. Experimental results demonstrate that NexusSplats achieves state-of-the-art rendering quality and reduces the number of total parameters by 65.4%, leading to 2.7$ imes$ faster reconstruction.

Problem

• Efficient 3D scene reconstruction under complex lighting
• Improved occlusion handling in unstructured real-world scenes
• Reduced computational cost and faster reconstruction times

Innovation

• Hierarchical light decoupling for efficient appearance learning
• Structure-aware occlusion handling between 3D and 2D structures
• Reduced parameters by 65.4% for faster reconstruction