See In Detail: Enhancing Sparse-view 3D Gaussian Splatting with Local Depth and Semantic Regularization

2025-01-20

AI Summary

To address geometric distortion, detail loss, and inter-view inconsistency in 3D Gaussian Splatting (3DGS) reconstruction under sparse-view settings, this paper proposes a novel method integrating local depth constraints with multi-view semantic consistency regularization. We innovatively incorporate self-supervised visual features extracted by DINO-ViT into the 3DGS optimization framework to formulate a semantic consistency loss—marking the first such integration in 3DGS. Additionally, we introduce a local depth smoothness regularization term to mitigate ill-posedness arising from sparse input views. Evaluated on the LLFF dataset, our method achieves a 0.4 dB PSNR improvement over baseline methods. It significantly suppresses geometric distortions, enhances structural accuracy and texture fidelity in novel-view synthesis, and improves the robustness and generalization capability of 3D reconstruction under sparse-view conditions.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Local Search
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

3D Gaussian Splatting (3DGS) has shown remarkable performance in novel view synthesis. However, its rendering quality deteriorates with sparse inphut views, leading to distorted content and reduced details. This limitation hinders its practical application. To address this issue, we propose a sparse-view 3DGS method. Given the inherently ill-posed nature of sparse-view rendering, incorporating prior information is crucial. We propose a semantic regularization technique, using features extracted from the pretrained DINO-ViT model, to ensure multi-view semantic consistency. Additionally, we propose local depth regularization, which constrains depth values to improve generalization on unseen views. Our method outperforms state-of-the-art novel view synthesis approaches, achieving up to 0.4dB improvement in terms of PSNR on the LLFF dataset, with reduced distortion and enhanced visual quality.

Problem

• 3D image technology
• view synthesis quality
• limited original views

Innovation

• 3D Gaussian Splattering
• Semantic Regularization
• Local Depth Regularization