GSFix3D: Diffusion-Guided Repair of Novel Views in Gaussian Splatting

2025-08-20

AI Summary

To address the degradation in reconstruction quality of 3D Gaussian Splatting under extreme novel viewpoints or sparse observations, this paper proposes GSFixer—a text-free, latent diffusion model (LDM)-guided inpainting framework. Methodologically, it introduces a joint 3D Gaussian–grid representation, integrates random masking augmentation with diffusion-guided completion of missing regions, and applies lightweight, task-specific fine-tuning of the LDM. Crucially, GSFixer achieves generalized defect correction using only geometric scene priors—without textual prompts—for the first time. Evaluated on multiple challenging benchmarks, it attains state-of-the-art performance. Moreover, it requires only a small amount of real-world calibration data for adaptation and demonstrates strong robustness to pose estimation errors in practical scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Recent developments in 3D Gaussian Splatting have significantly enhanced novel view synthesis, yet generating high-quality renderings from extreme novel viewpoints or partially observed regions remains challenging. Meanwhile, diffusion models exhibit strong generative capabilities, but their reliance on text prompts and lack of awareness of specific scene information hinder accurate 3D reconstruction tasks. To address these limitations, we introduce GSFix3D, a novel framework that improves the visual fidelity in under-constrained regions by distilling prior knowledge from diffusion models into 3D representations, while preserving consistency with observed scene details. At its core is GSFixer, a latent diffusion model obtained via our customized fine-tuning protocol that can leverage both mesh and 3D Gaussians to adapt pretrained generative models to a variety of environments and artifact types from different reconstruction methods, enabling robust novel view repair for unseen camera poses. Moreover, we propose a random mask augmentation strategy that empowers GSFixer to plausibly inpaint missing regions. Experiments on challenging benchmarks demonstrate that our GSFix3D and GSFixer achieve state-of-the-art performance, requiring only minimal scene-specific fine-tuning on captured data. Real-world test further confirms its resilience to potential pose errors. Our code and data will be made publicly available. Project page: https://gsfix3d.github.io.

Problem

• Improving novel view synthesis from extreme viewpoints
• Enhancing 3D reconstruction with diffusion model knowledge
• Repairing missing regions in Gaussian Splatting renderings

Innovation

• Diffusion-guided repair for Gaussian Splatting views
• Custom fine-tuning protocol for latent diffusion model
• Random mask augmentation strategy for inpainting