D3DR: Lighting-Aware Object Insertion in Gaussian Splatting

2025-03-09

AI Summary

To address the challenge of inconsistent lighting and shadowing when inserting objects into 3D Gaussian Splatting (3DGS) scenes—leading to unnatural visual integration—this paper proposes the first lighting-aware object insertion framework tailored for 3DGS. Methodologically, it synergistically combines differentiable rendering with diffusion model implicit priors, introducing Delta Denoising Score (DDS) as a novel score-based objective for lighting correction. Gaussian parameters are jointly optimized via personalized fine-tuning and score-guided optimization. Compared to existing approaches, our method achieves measurable improvements of +0.5 PSNR and +0.15 SSIM in relit reconstruction quality, while significantly enhancing shadow alignment and material consistency. The result is physically plausible, visually seamless object editing within 3DGS scenes.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Gaussian Splatting has become a popular technique for various 3D Computer Vision tasks, including novel view synthesis, scene reconstruction, and dynamic scene rendering. However, the challenge of natural-looking object insertion, where the object's appearance seamlessly matches the scene, remains unsolved. In this work, we propose a method, dubbed D3DR, for inserting a 3DGS-parametrized object into 3DGS scenes while correcting its lighting, shadows, and other visual artifacts to ensure consistency, a problem that has not been successfully addressed before. We leverage advances in diffusion models, which, trained on real-world data, implicitly understand correct scene lighting. After inserting the object, we optimize a diffusion-based Delta Denoising Score (DDS)-inspired objective to adjust its 3D Gaussian parameters for proper lighting correction. Utilizing diffusion model personalization techniques to improve optimization quality, our approach ensures seamless object insertion and natural appearance. Finally, we demonstrate the method's effectiveness by comparing it to existing approaches, achieving 0.5 PSNR and 0.15 SSIM improvements in relighting quality.

Problem

• Seamless object insertion in 3D scenes
• Lighting and shadow correction for 3D objects
• Improving visual consistency using diffusion models

Innovation

• Uses diffusion models for lighting correction
• Optimizes 3D Gaussian parameters with DDS
• Applies diffusion model personalization techniques