LightHarmony3D: Harmonizing Illumination and Shadows for Object Insertion in 3D Gaussian Splatting

2026-03-30

AI Summary

This work addresses the challenge of maintaining photometric consistency and multi-view coherence when inserting external mesh objects into 3D Gaussian Splatting (3DGS) scenes. To this end, we propose LightHarmony3D, a novel framework that introduces, for the first time, a generative module capable of predicting a full 360° HDR environment map at the insertion location through a single forward pass. By integrating this predicted lighting with physically based rendering, our method enables realistic object insertion while preserving scene illumination fidelity. We also present the first dedicated benchmark for evaluating mesh insertion in 3DGS. Extensive experiments on multiple real-world datasets demonstrate that LightHarmony3D significantly outperforms existing approaches, achieving state-of-the-art performance in both lighting realism and multi-view consistency.

Abstract

3D Gaussian Splatting (3DGS) enables high-fidelity reconstruction of scene geometry and appearance. Building on this capability, inserting external mesh objects into reconstructed 3DGS scenes enables interactive editing and content augmentation for immersive applications such as AR/VR, virtual staging, and digital content creation. However, achieving physically consistent lighting and shadows for mesh insertion remains challenging, as it requires accurate scene illumination estimation and multi-view consistent rendering. To address this challenge, we present LightHarmony3D, a novel framework for illumination-consistent mesh insertion in 3DGS scenes. Central to our approach is our proposed generative module that predicts a full 360° HDR environment map at the insertion location via a single forward pass. By leveraging generative priors instead of iterative optimization, our method efficiently captures dominant scene illumination and enables physically grounded shading and shadows for inserted meshes while maintaining multi-view coherence. Furthermore, we introduce the first dedicated benchmark for mesh insertion in 3DGS, providing a standardized evaluation framework for assessing lighting consistency and photorealism. Extensive experiments across multiple real-world reconstruction datasets demonstrate that LightHarmony3D achieves state-of-the-art realism and multi-view consistency.

Problem

• 3D Gaussian Splatting
• object insertion
• illumination consistency
• shadow rendering
• multi-view coherence

Innovation

• 3D Gaussian Splatting
• illumination estimation
• HDR environment map
• mesh insertion
• multi-view consistency