3D Scene-Camera Representation with Joint Camera Photometric Optimization

2025-06-25

AI Summary

In multi-view 3D reconstruction, photometric distortions—such as vignetting and lens contamination—degrade scene representation fidelity. To address this, we propose a novel joint optimization framework that simultaneously learns both the 3D radiance field and camera-specific photometric models. Our approach introduces an explicit parametrization of intrinsic and extrinsic photometric effects, tightly coupling differentiable rendering with depth-based regularization to disentangle imaging distortions from underlying scene geometry and appearance. During training, radiance field parameters and photometric correction parameters are co-optimized, effectively suppressing the adverse impact of imaging noise on reconstruction accuracy. Experiments demonstrate that our method significantly improves reconstruction fidelity and robustness under photometric degradation, outperforming conventional methods that ignore photometric modeling. It achieves state-of-the-art performance across multiple standard benchmarks.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Representing scenes from multi-view images is a crucial task in computer vision with extensive applications. However, inherent photometric distortions in the camera imaging can significantly degrade image quality. Without accounting for these distortions, the 3D scene representation may inadvertently incorporate erroneous information unrelated to the scene, diminishing the quality of the representation. In this paper, we propose a novel 3D scene-camera representation with joint camera photometric optimization. By introducing internal and external photometric model, we propose a full photometric model and corresponding camera representation. Based on simultaneously optimizing the parameters of the camera representation, the proposed method effectively separates scene-unrelated information from the 3D scene representation. Additionally, during the optimization of the photometric parameters, we introduce a depth regularization to prevent the 3D scene representation from fitting scene-unrelated information. By incorporating the camera model as part of the mapping process, the proposed method constructs a complete map that includes both the scene radiance field and the camera photometric model. Experimental results demonstrate that the proposed method can achieve high-quality 3D scene representations, even under conditions of imaging degradation, such as vignetting and dirt.

Problem

• Addressing photometric distortions in multi-view 3D scene representation
• Separating scene-unrelated information from 3D reconstructions
• Optimizing camera photometric models alongside scene radiance fields

Innovation

• Joint camera photometric optimization for 3D scenes
• Full photometric model with internal and external parameters
• Depth regularization prevents scene-unrelated information fitting