View-Consistent 3D Editing with Gaussian Splatting

2024-03-18

European Conference on Computer Vision

AI Summary

This work addresses the problem of mode collapse and geometric artifacts in 3D Gaussian Splatting (3DGS) multi-view editing, caused by cross-view inconsistency in diffusion-generated 2D guidance maps. To this end, we propose VcEdit—a novel end-to-end view-consistent editing framework. Methodologically, we introduce two key modules: the Cross-Attention Consistency Module (CACM) and the Editing Consistency Module (ECM), which explicitly model and enforce both geometric and semantic consistency across views within a render-edit-reoptimize loop. Notably, VcEdit is the first to incorporate iterative cross-view consistency regularization into the 3DGS editing pipeline. Experiments demonstrate that VcEdit significantly improves multi-view consistency and geometric fidelity of edited 3DGS models, effectively eliminating artifacts in complex scenes while enabling high-quality, fine-grained local 3D content editing.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

The advent of 3D Gaussian Splatting (3DGS) has revolutionized 3D editing, offering efficient, high-fidelity rendering and enabling precise local manipulations. Currently, diffusion-based 2D editing models are harnessed to modify multi-view rendered images, which then guide the editing of 3DGS models. However, this approach faces a critical issue of multi-view inconsistency, where the guidance images exhibit significant discrepancies across views, leading to mode collapse and visual artifacts of 3DGS. To this end, we introduce View-consistent Editing (VcEdit), a novel framework that seamlessly incorporates 3DGS into image editing processes, ensuring multi-view consistency in edited guidance images and effectively mitigating mode collapse issues. VcEdit employs two innovative consistency modules: the Cross-attention Consistency Module and the Editing Consistency Module, both designed to reduce inconsistencies in edited images. By incorporating these consistency modules into an iterative pattern, VcEdit proficiently resolves the issue of multi-view inconsistency, facilitating high-quality 3DGS editing across a diverse range of scenes. Further video results are shown in http://vcedit.github.io.

Problem

• Ensures multi-view consistency in 3D editing
• Mitigates mode collapse in Gaussian Splatting
• Reduces visual artifacts in 3DGS models

Innovation

• View-consistent Editing framework
• Cross-attention Consistency Module
• Editing Consistency Module