Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting

2025-06-05

AI Summary

In feedforward deep graph-driven 3D Gaussian Splatting (3DGS), depth discontinuities at object boundaries cause point cloud fragmentation and degraded rendering quality. To address this, we propose a geometry-aware depth optimization framework: leveraging a pre-trained Transformer to generate a structurally coherent point map as geometric prior, we introduce a novel PM-Loss regularization term that explicitly enforces geometric smoothness in depth estimation near boundaries. Our method jointly optimizes the point map and depth map, integrates differentiable inverse projection, and embeds seamlessly into the 3DGS rendering pipeline. Experiments across diverse architectures and scenes demonstrate significant improvements in rendering fidelity—higher PSNR and SSIM, lower LPIPS—along with markedly enhanced point cloud completeness and geometric fidelity. This approach overcomes the inherent limitation of conventional depth maps in modeling discontinuous regions, establishing a new state-of-the-art for boundary-aware depth-guided 3DGS reconstruction.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT

Abstract

Depth maps are widely used in feed-forward 3D Gaussian Splatting (3DGS) pipelines by unprojecting them into 3D point clouds for novel view synthesis. This approach offers advantages such as efficient training, the use of known camera poses, and accurate geometry estimation. However, depth discontinuities at object boundaries often lead to fragmented or sparse point clouds, degrading rendering quality -- a well-known limitation of depth-based representations. To tackle this issue, we introduce PM-Loss, a novel regularization loss based on a pointmap predicted by a pre-trained transformer. Although the pointmap itself may be less accurate than the depth map, it effectively enforces geometric smoothness, especially around object boundaries. With the improved depth map, our method significantly improves the feed-forward 3DGS across various architectures and scenes, delivering consistently better rendering results. Our project page: https://aim-uofa.github.io/PMLoss

Problem

• Address depth discontinuities in 3D Gaussian Splatting
• Improve point cloud quality at object boundaries
• Enhance rendering accuracy with geometric smoothness

Innovation

• Introduces PM-Loss for geometric smoothness
• Uses pre-trained transformer for pointmap prediction
• Improves depth maps in 3D Gaussian Splatting