RayGauss: Volumetric Gaussian-Based Ray Casting for Photorealistic Novel View Synthesis

2024-08-06

arXiv.org

AI Summary

Differentiable ray casting on irregularly distributed Gaussian kernels suffers from severe rendering artifacts (e.g., splatter) and low efficiency in novel-view synthesis. Method: We propose a voxelized Gaussian modeling framework that establishes, for the first time, a physically consistent, density-radiance decoupled differentiable ray casting model. Our approach introduces hierarchical voxel slab integration with BVH acceleration for efficient and accurate volumetric rendering; jointly represents full-spectrum color using spherical Gaussians and spherical harmonics; and co-optimizes Gaussian geometry and radiance properties. Contribution/Results: On the Blender dataset, our method achieves real-time inference at 25 FPS with reasonable training efficiency. It significantly outperforms state-of-the-art methods in PSNR and SSIM while effectively suppressing splatter artifacts. This work bridges a critical gap between differentiable ray casting and Gaussian-based scene representation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Differentiable volumetric rendering-based methods made significant progress in novel view synthesis. On one hand, innovative methods have replaced the Neural Radiance Fields (NeRF) network with locally parameterized structures, enabling high-quality renderings in a reasonable time. On the other hand, approaches have used differentiable splatting instead of NeRF's ray casting to optimize radiance fields rapidly using Gaussian kernels, allowing for fine adaptation to the scene. However, differentiable ray casting of irregularly spaced kernels has been scarcely explored, while splatting, despite enabling fast rendering times, is susceptible to clearly visible artifacts. Our work closes this gap by providing a physically consistent formulation of the emitted radiance c and density {sigma}, decomposed with Gaussian functions associated with Spherical Gaussians/Harmonics for all-frequency colorimetric representation. We also introduce a method enabling differentiable ray casting of irregularly distributed Gaussians using an algorithm that integrates radiance fields slab by slab and leverages a BVH structure. This allows our approach to finely adapt to the scene while avoiding splatting artifacts. As a result, we achieve superior rendering quality compared to the state-of-the-art while maintaining reasonable training times and achieving inference speeds of 25 FPS on the Blender dataset. Project page with videos and code: https://raygauss.github.io/

Problem

• Improves novel view synthesis quality
• Enables differentiable ray casting
• Avoids splatting artifacts effectively

Innovation

• Differentiable ray casting
• Gaussian-based volumetric rendering
• BVH structure integration