CoIn: Comprehensive 2D-3D Inpainting with Gaussian Splatting Guidance

2026-06-25

AI Summary

This work addresses the challenge of reconstructing missing regions in 3D scenes caused by occlusions or limited viewpoints. While existing approaches typically rely on precise multi-view segmentation masks and support only object removal, this paper presents the first 3D scene inpainting framework capable of both removing and inserting objects under arbitrarily shaped masks. The method integrates 2D image inpainting with 3D Gaussian Splatting through a bidirectional 2D–3D guidance mechanism to ensure multi-view consistency. Key innovations include reference-adaptive Gaussian splatting, a feature attention module, and a texture-enhanced discriminator, combined with diffusion-based inpainting and reference feature warping. This approach significantly outperforms state-of-the-art methods in both photorealism and geometric consistency.

Abstract

3D scene inpainting is essential for reconstructing areas corrupted by occlusions or limited viewpoints. While recent methods leverage Gaussian Splatting (GS) for efficient 3D editing, they often depend on precise multi-view segmentation masks and are inherently constrained to object removal tasks. We propose CoIn, a novel framework that bridges 2D inpainting models and 3DGS through a multi-stage consistency pipeline. Our approach first generates initial inpainted images using a diffusion model, enabling the use of arbitrary-shaped masks and diverse tasks like object insertion. We then introduce Reference Adaptive GS with Feature Attention to reconstruct a coarse 3D scene by adaptively weighing towards a reference view (2D -> 3D). This 3D representation provides geometric guidance to the diffusion process via GS-based Reference Feature Warping, ensuring multi-view consistency (3D -> 2D). Finally, a Texture-Enhancing Discriminator refines the 3D scene to achieve high photometric realism (2D -> 3D). Experiments show that CoIn, effectively leveraging bidirectional information flow, achieves state-of-the-art performance and effectively handles both object removal and object insertion with flexible mask input.

Problem

• 3D scene inpainting
• object removal
• object insertion
• multi-view consistency
• mask flexibility

Innovation

• Gaussian Splatting
• 3D Inpainting
• Diffusion Model
• Multi-view Consistency
• Bidirectional 2D-3D Integration