CoCoGaussian: Leveraging Circle of Confusion for Gaussian Splatting from Defocused Images

2024-12-20

arXiv.org

AI Summary

Existing 3D Gaussian Splatting (3DGS) methods struggle to achieve high-fidelity reconstruction from defocused images due to limited depth-of-field in real-world scenes. To address this, we propose a Circle-of-Confusion (CoC)-aware 3DGS framework that for the first time integrates a physically grounded imaging model into 3DGS: it jointly models CoC diameter using scene depth and a learnable aperture parameter, and introduces a learnable scaling factor to enhance robustness on reflective/refractive surfaces. Our method unifies photogeometric constraints, differentiable CoC-aware rendering, depth-guided multi-Gaussian distribution generation, and adaptive scale optimization. Evaluated on both synthetic and real-world datasets, our approach achieves state-of-the-art performance, significantly improving 3D reconstruction accuracy and novel-view synthesis quality from defocused inputs.

Technology Category

• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT

Abstract

3D Gaussian Splatting (3DGS) has attracted significant attention for its high-quality novel view rendering, inspiring research to address real-world challenges. While conventional methods depend on sharp images for accurate scene reconstruction, real-world scenarios are often affected by defocus blur due to finite depth of field, making it essential to account for realistic 3D scene representation. In this study, we propose CoCoGaussian, a Circle of Confusion-aware Gaussian Splatting that enables precise 3D scene representation using only defocused images. CoCoGaussian addresses the challenge of defocus blur by modeling the Circle of Confusion (CoC) through a physically grounded approach based on the principles of photographic defocus. Exploiting 3D Gaussians, we compute the CoC diameter from depth and learnable aperture information, generating multiple Gaussians to precisely capture the CoC shape. Furthermore, we introduce a learnable scaling factor to enhance robustness and provide more flexibility in handling unreliable depth in scenes with reflective or refractive surfaces. Experiments on both synthetic and real-world datasets demonstrate that CoCoGaussian achieves state-of-the-art performance across multiple benchmarks.

Problem

• Enables 3D scene reconstruction from defocused images
• Models Circle of Confusion for accurate blur handling
• Improves robustness for reflective or refractive surfaces

Innovation

• Uses Circle of Confusion for Gaussian Splatting
• Models defocus blur with photographic principles
• Introduces learnable scaling for depth robustness