ResGS: Residual Densification of 3D Gaussian for Efficient Detail Recovery

2024-12-10

arXiv.org

AI Summary

3D Gaussian Splatting (3D-GS) suffers from detail loss and geometric incompleteness in novel-view synthesis, primarily due to its fixed densification strategy, which fails to simultaneously ensure geometric coverage and fine-detail recovery. To address this, we propose an adaptive residual splitting densification mechanism: (i) learnable scaled Gaussian residuals are introduced to enhance both geometric completion and high-frequency detail modeling; (ii) a Gaussian image pyramid with progressive supervision and coarse-to-fine selection enforces multi-scale collaborative optimization. Our method establishes the first densification paradigm supporting both residual modeling and adaptive splitting—fully plug-and-play without modifying the backbone architecture. Evaluated on multiple benchmarks, it achieves state-of-the-art rendering quality, improving detail fidelity by +1.28 dB PSNR and geometric completeness by −14.3% Chamfer distance, thereby overcoming the fundamental trade-off limitations of conventional densification strategies.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Recently, 3D Gaussian Splatting (3D-GS) has prevailed in novel view synthesis, achieving high fidelity and efficiency. However, it often struggles to capture rich details and complete geometry. Our analysis reveals that the 3D-GS densification operation lacks adaptiveness and faces a dilemma between geometry coverage and detail recovery. To address this, we introduce a novel densification operation, residual split, which adds a downscaled Gaussian as a residual. Our approach is capable of adaptively retrieving details and complementing missing geometry. To further support this method, we propose a pipeline named ResGS. Specifically, we integrate a Gaussian image pyramid for progressive supervision and implement a selection scheme that prioritizes the densification of coarse Gaussians over time. Extensive experiments demonstrate that our method achieves SOTA rendering quality. Consistent performance improvements can be achieved by applying our residual split on various 3D-GS variants, underscoring its versatility and potential for broader application in 3D-GS-based applications.

Problem

• 3D-GS lacks adaptiveness in detail recovery
• Geometry coverage vs detail recovery dilemma
• ResGS improves rendering quality adaptively

Innovation

• Residual split for adaptive detail recovery
• Gaussian image pyramid for progressive supervision
• Prioritized densification of coarse Gaussians