Fovea Stacking: Imaging with Dynamic Localized Aberration Correction

2025-05-31

AI Summary

Off-axis aberrations severely degrade image quality in miniaturized optical imaging systems, and conventional software-based correction methods fail to address them effectively. Method: This paper proposes a dynamic, localized aberration correction approach leveraging a deformable phase plate (DPP). We introduce the novel “foveated stacking” paradigm, enabling full-field, aberration-free reconstruction via adaptive multi-focus image stacking in focal space. The method integrates differentiable optical modeling, neural-network-based nonlinear compensation, and joint optimization to support real-time, eye- or target-driven dynamic focusing. Contribution/Results: Compared with conventional focus stacking, our approach significantly enhances off-axis image sharpness and extends depth of field. It achieves, for the first time, hardware–software co-optimized real-time foveated video imaging. The system demonstrates strong applicability in scenarios demanding both ultra-compact form factors and high optical fidelity—such as surveillance and VR displays.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Hardware-aware ML
• Search and Optimization: Non-convex Optimization

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The desire for cameras with smaller form factors has recently lead to a push for exploring computational imaging systems with reduced optical complexity such as a smaller number of lens elements. Unfortunately such simplified optical systems usually suffer from severe aberrations, especially in off-axis regions, which can be difficult to correct purely in software. In this paper we introduce Fovea Stacking, a new type of imaging system that utilizes emerging dynamic optical components called deformable phase plates (DPPs) for localized aberration correction anywhere on the image sensor. By optimizing DPP deformations through a differentiable optical model, off-axis aberrations are corrected locally, producing a foveated image with enhanced sharpness at the fixation point - analogous to the eye's fovea. Stacking multiple such foveated images, each with a different fixation point, yields a composite image free from aberrations. To efficiently cover the entire field of view, we propose joint optimization of DPP deformations under imaging budget constraints. Due to the DPP device's non-linear behavior, we introduce a neural network-based control model for improved alignment between simulation-hardware performance. We further demonstrated that for extended depth-of-field imaging, fovea stacking outperforms traditional focus stacking in image quality. By integrating object detection or eye-tracking, the system can dynamically adjust the lens to track the object of interest-enabling real-time foveated video suitable for downstream applications such as surveillance or foveated virtual reality displays.

Problem

• Corrects off-axis aberrations in simplified optical systems
• Uses deformable phase plates for localized aberration correction
• Enables real-time foveated video for dynamic applications

Innovation

• Uses deformable phase plates for localized correction
• Optimizes DPP deformations via differentiable optical model
• Employs neural network for simulation-hardware alignment