2D Gaussian Splatting with Semantic Alignment for Image Inpainting

2025-09-02

AI Summary

This work addresses the challenge of simultaneously preserving local coherence and global semantic consistency in image inpainting. We propose the first differentiable inpainting framework based on a 2D Gaussian lattice, modeling missing regions as a continuous Gaussian coefficient field. Our method combines tiled rasterization with differentiable rendering for efficient, pixel-level reconstruction. To enforce semantic alignment, we incorporate guidance from pre-trained DINO features, significantly improving structural plausibility and contextual consistency—especially for large missing regions. Unlike conventional grid-based or implicit representations, our paradigm offers both explicit geometric controllability and end-to-end optimizability. Evaluated on standard benchmarks, our approach achieves state-of-the-art performance in PSNR, LPIPS, and perceptual quality, while maintaining high inference efficiency. This demonstrates a superior balance among inpainting completeness, semantic fidelity, and computational efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Local Search
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Security and Privacy: Data transparency and provenance

Abstract

Gaussian Splatting (GS), a recent technique for converting discrete points into continuous spatial representations, has shown promising results in 3D scene modeling and 2D image super-resolution. In this paper, we explore its untapped potential for image inpainting, which demands both locally coherent pixel synthesis and globally consistent semantic restoration. We propose the first image inpainting framework based on 2D Gaussian Splatting, which encodes incomplete images into a continuous field of 2D Gaussian splat coefficients and reconstructs the final image via a differentiable rasterization process. The continuous rendering paradigm of GS inherently promotes pixel-level coherence in the inpainted results. To improve efficiency and scalability, we introduce a patch-wise rasterization strategy that reduces memory overhead and accelerates inference. For global semantic consistency, we incorporate features from a pretrained DINO model. We observe that DINO's global features are naturally robust to small missing regions and can be effectively adapted to guide semantic alignment in large-mask scenarios, ensuring that the inpainted content remains contextually consistent with the surrounding scene. Extensive experiments on standard benchmarks demonstrate that our method achieves competitive performance in both quantitative metrics and perceptual quality, establishing a new direction for applying Gaussian Splatting to 2D image processing.

Problem

• Applying Gaussian Splatting to image inpainting tasks
• Ensuring local coherence and global semantic consistency
• Handling large-mask scenarios with contextual alignment

Innovation

• 2D Gaussian Splatting for image inpainting
• Patch-wise rasterization for efficiency
• DINO features for semantic alignment