Dual-Camera All-in-Focus Neural Radiance Fields

2025-01-30

IEEE Transactions on Pattern Analysis and Machine Intelligence

AI Summary

To address the global defocus blur and lack of sharp geometric and textural references in single-camera NeRF caused by fixed focal length, this paper proposes the first full-depth-of-field NeRF reconstruction method leveraging smartphone dual-camera systems (primary + ultra-wide). The ultra-wide camera provides robust geometric priors via its large depth of field, while the primary camera preserves high-resolution texture. We design a learnable defocus-aware fusion module and a dynamic defocus map prediction mechanism, integrated with spatial deformation alignment, cross-camera color matching, and differentiable defocus modeling for end-to-end full-depth-of-field radiance field reconstruction. Evaluated on our newly collected dual-camera dataset, our method significantly outperforms single-camera NeRF baselines. It enables arbitrary focal-plane refocusing, controllable blur intensity, and beam-splitter effects, generating high-fidelity novel-view images.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

We present the first framework capable of synthesizing the all-in-focus neural radiance field (NeRF) from inputs without manual refocusing. Without refocusing, the camera will automatically focus on the fixed object for all views, and current NeRF methods typically using one camera fail due to the consistent defocus blur and a lack of sharp reference. To restore the all-in-focus NeRF, we introduce the dual-camera from smartphones, where the ultra-wide camera has a wider depth-of-field (DoF) and the main camera possesses a higher resolution. The dual camera pair saves the high-fidelity details from the main camera and uses the ultra-wide camera’s deep DoF as reference for all-in-focus restoration. To this end, we first implement spatial warping and color matching to align the dual camera, followed by a defocus-aware fusion module with learnable defocus parameters to predict a defocus map and fuse the aligned camera pair. We also build a multi-view dataset that includes image pairs of the main and ultra-wide cameras in a smartphone. Extensive experiments on this dataset verify that our solution, termed DC-NeRF, can produce high-quality all-in-focus novel views and compares favorably against strong baselines quantitatively and qualitatively. We further show DoF applications of DC-NeRF with adjustable blur intensity and focal plane, including refocusing and split diopter.

Problem

• Synthesize all-in-focus NeRF without manual refocusing
• Overcome defocus blur in single-camera NeRF methods
• Leverage dual-camera data for depth-of-field restoration

Innovation

• Uses dual-camera setup for all-in-focus NeRF
• Implements defocus-aware fusion with learnable parameters
• Builds multi-view dataset for smartphone cameras