Towards Single-Lens Controllable Depth-of-Field Imaging via Depth-Aware Point Spread Functions

2024-09-15

AI Summary

This work addresses two fundamental bottlenecks in mobile lightweight minimalist optical systems (MOS): severe optical aberrations and uncontrollable depth of field (DoF). To this end, we propose the first depth-aware controllable DoF imaging framework (DCDI). Methodologically: (1) we introduce a depth-adaptive degradation training (DA²T) strategy to jointly model optical aberrations and scene depth; (2) we design a memory-efficient omni-lens-field point spread function (PSF) representation that captures spatially varying aberrations across the entire lens field; and (3) we release DAMOS—the first depth-aware aberration dataset for MOS. Experiments demonstrate that DCDI significantly improves aberration correction accuracy in both simulation and real-world settings, while enabling arbitrary-DoF rendering from a single lens—achieving visual quality comparable to high-end multi-lens systems. This work establishes a new paradigm for lightweight computational photography.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Search and Optimization: Learning to Search
• Machine Learning: Hardware-aware ML

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Controllable Depth-of-Field (DoF) imaging commonly produces amazing visual effects based on heavy and expensive high-end lenses. However, confronted with the increasing demand for mobile scenarios, it is desirable to achieve a lightweight solution with Minimalist Optical Systems (MOS). This work centers around two major limitations of MOS, i.e., the severe optical aberrations and uncontrollable DoF, for achieving single-lens controllable DoF imaging via computational methods. A Depth-aware Controllable DoF Imaging (DCDI) framework is proposed equipped with All-in-Focus (AiF) aberration correction and monocular depth estimation, where the recovered image and corresponding depth map are utilized to produce imaging results under diverse DoFs of any high-end lens via patch-wise convolution. To address the depth-varying optical degradation, we introduce a Depth-aware Degradation-adaptive Training (DA2T) scheme. At the dataset level, a Depth-aware Aberration MOS (DAMOS) dataset is established based on the simulation of Point Spread Functions (PSFs) under different object distances. Additionally, we design two plug-and-play depth-aware mechanisms to embed depth information into the aberration image recovery for better tackling depth-aware degradation. Furthermore, we propose a storage-efficient Omni-Lens-Field model to represent the 4D PSF library of various lenses. With the predicted depth map, recovered image, and depth-aware PSF map inferred by Omni-Lens-Field, single-lens controllable DoF imaging is achieved. Comprehensive experimental results demonstrate that the proposed framework enhances the recovery performance, and attains impressive single-lens controllable DoF imaging results, providing a seminal baseline for this field. The source code and the established dataset will be publicly available at https://github.com/XiaolongQian/DCDI.

Problem

• Achieve single-lens controllable depth-of-field
• Address optical aberrations in minimalist systems
• Enhance imaging via depth-aware degradation adaptation

Innovation

• Depth-aware Controllable DoF Imaging
• Depth-aware Degradation-adaptive Training
• Omni-Lens-Field PSF library