3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes

2024-11-22

arXiv.org

AI Summary

3D Gaussian Splatting (3DGS) suffers from inherent limitations in modeling sharp edges, planar surfaces, and geometric compactness—manifesting as edge blurring, “float-in-air” artifacts, irregular spatial distribution around surfaces, and reliance on hand-crafted regularization. To address these, we propose constructing geometrically and semantically explicit radiance fields using differentiable 3D smooth convex bodies as primitives. This work pioneers the replacement of Gaussians with smooth convex bodies as fundamental radiance field units, inherently achieving tight surface-aligned distribution without explicit regularization while preserving edge sharpness and volumetric density representation. Technically, our approach integrates differentiable convex body parameterization, multi-view self-supervised optimization, and a custom CUDA rasterizer enabling efficient forward rendering and gradient backpropagation. On benchmarks including Mip-NeRF360, our method achieves up to +0.81 PSNR gain and −0.026 LPIPS reduction, matches 3DGS rendering speed, and significantly reduces primitive count.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Non-convex Optimization
• Machine Learning: Learning with Manifolds

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance

Abstract

Recent advances in radiance field reconstruction, such as 3D Gaussian Splatting (3DGS), have achieved high-quality novel view synthesis and fast rendering by representing scenes with compositions of Gaussian primitives. However, 3D Gaussians present several limitations for scene reconstruction. Accurately capturing hard edges is challenging without significantly increasing the number of Gaussians, creating a large memory footprint. Moreover, they struggle to represent flat surfaces, as they are diffused in space. Without hand-crafted regularizers, they tend to disperse irregularly around the actual surface. To circumvent these issues, we introduce a novel method, named 3D Convex Splatting (3DCS), which leverages 3D smooth convexes as primitives for modeling geometrically-meaningful radiance fields from multi-view images. Smooth convex shapes offer greater flexibility than Gaussians, allowing for a better representation of 3D scenes with hard edges and dense volumes using fewer primitives. Powered by our efficient CUDA-based rasterizer, 3DCS achieves superior performance over 3DGS on benchmarks such as Mip-NeRF360, Tanks and Temples, and Deep Blending. Specifically, our method attains an improvement of up to 0.81 in PSNR and 0.026 in LPIPS compared to 3DGS while maintaining high rendering speeds and reducing the number of required primitives. Our results highlight the potential of 3D Convex Splatting to become the new standard for high-quality scene reconstruction and novel view synthesis. Project page: convexsplatting.github.io.

Problem

• Improving hard edge representation in radiance fields
• Reducing memory footprint in 3D scene reconstruction
• Enhancing flat surface modeling with fewer primitives

Innovation

• Uses 3D smooth convexes as primitives
• Efficient CUDA-based rasterizer for rendering
• Improves PSNR and LPIPS over 3DGS