Localized Gaussian Splatting Editing with Contextual Awareness

2024-07-31

IEEE Workshop/Winter Conference on Applications of Computer Vision

AI Summary

Existing text-guided 3D object insertion/replacement methods neglect background illumination modeling, resulting in inconsistent lighting between synthesized objects and the scene. To address this illumination inconsistency challenge in 3D Gaussian Splatting (3DGS) scene editing, we propose a lighting-aware two-stage editing framework: first, depth-guided 2D diffusion inpainting on anchor views implicitly models global illumination; second, coarse-to-fine optimization jointly distills geometry and appearance. Our core contribution is Depth-guided Inpainting Score Distillation Sampling (DI-SDS), the first method to tightly couple 2D diffusion inpainting with Score Distillation Sampling (SDS) in 3DGS. DI-SDS enables high-fidelity, locally consistent edits with natural illumination and geometric accuracy—even under strong highlights and complex shadows—in real-world scenes. Extensive experiments demonstrate significant improvements over state-of-the-art text-driven 3D editing approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent advancements in text-guided 3D object generation using diffusion priors struggle with illumination inconsistencies when applied to scene editing tasks like object replacement or insertion. To address this, we propose an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS). Our method leverages state-of-the-art 2D diffusion inpainting [56] to handle global illumination context effectively. Specifically, we identify representative anchor views that capture scene-wide illumination, inpaint them using 2D diffusion models, and integrate the results into a coarse-to-fine 3DGS optimization process. In the fine step, we introduce Depth-guided Inpainting Score Distillation Sampling (DI-SDS) to refine geometry and texture details, capitalizing on the diversity of 2D priors. Our approach achieves locally precise edits with globally consistent illumination, demonstrating robustness in real scenes with highlights and shadows. Comparisons show superior results over state-of-the-art text-to-3D editing methods. Project page: https://corneliushsiao.github.io/GSLE.html.

Problem

• Achieving illumination-aware 3D object insertion and replacement tasks
• Bridging illumination mismatches between generated objects and background environments
• Enabling localized Gaussian splatting editing with contextual lighting consistency

Innovation

• Illumination-aware 3D Gaussian Splatting editing pipeline
• Anchor View Proposal algorithm for scene illumination
• Depth-guided Inpainting Score Distillation Sampling enhancement