MiraGe: Editable 2D Images using Gaussian Splatting

2024-10-02

arXiv.org

AI Summary

This work addresses the challenge of achieving high-quality, physically consistent, and semantically editable 2D image editing within 3D space. We propose a semantic-controllable image editing framework built upon 3D Gaussian representations. Methodologically, we introduce the first integration of specular reflection geometry modeling with planar-constrained Gaussian splatting to construct a differentiable and editable implicit 3D scene representation; this is further coupled with a physics engine to ensure illumination consistency, geometrically plausible occlusion, and physically grounded dynamics during interactive editing. Compared to conventional implicit neural representations (INRs) and non-editable Gaussian rendering approaches, our method overcomes the semantic-level manipulation bottleneck. It achieves state-of-the-art compression efficiency while significantly improving editing fidelity and 3D geometric consistency—establishing a novel paradigm for photorealistic image editing.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Implicit Neural Representations (INRs) approximate discrete data through continuous functions and are commonly used for encoding 2D images. Traditional image-based INRs employ neural networks to map pixel coordinates to RGB values, capturing shapes, colors, and textures within the network's weights. Recently, GaussianImage has been proposed as an alternative, using Gaussian functions instead of neural networks to achieve comparable quality and compression. Such a solution obtains a quality and compression ratio similar to classical INR models but does not allow image modification. In contrast, our work introduces a novel method, MiraGe, which uses mirror reflections to perceive 2D images in 3D space and employs flat-controlled Gaussians for precise 2D image editing. Our approach improves the rendering quality and allows realistic image modifications, including human-inspired perception of photos in the 3D world. Thanks to modeling images in 3D space, we obtain the illusion of 3D-based modification in 2D images. We also show that our Gaussian representation can be easily combined with a physics engine to produce physics-based modification of 2D images. Consequently, MiraGe allows for better quality than the standard approach and natural modification of 2D images

Problem

• Enables precise 2D image editing
• Improves rendering quality using 3D space
• Allows physics-based modifications in 2D images

Innovation

• Uses mirror reflections for 3D perception
• Employs flat-controlled Gaussians for editing
• Combines with physics engine for modifications