Learned Off-aperture Encoding for Wide Field-of-view RGBD Imaging

2025-07-30

AI Summary

Existing end-to-end imaging systems struggle to simultaneously achieve high image fidelity and accurate depth sensing over wide fields-of-view (FoV), primarily because conventional aperture-plane coding only enables global wavefront modulation and suffers from inaccurate off-axis aberration modeling. This work introduces “off-aperture coding”—a novel paradigm that places a diffractive optical element (DOE) at a non-aperture plane to enable spatially localized and decoupled wavefront control. By integrating differentiable ray tracing with rigorous wave-optical modeling, we formulate a hybrid refractive-diffractive system amenable to joint RGB-D end-to-end optimization. Quantitative evaluation shows a PSNR improvement of over 5 dB at a 45° FoV. A physical prototype demonstrates simultaneous high-fidelity reconstruction of color and depth maps within a 28° FoV, validating both the efficacy and practical feasibility of the proposed approach.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

End-to-end (E2E) designed imaging systems integrate coded optical designs with decoding algorithms to enhance imaging fidelity for diverse visual tasks. However, existing E2E designs encounter significant challenges in maintaining high image fidelity at wide fields of view, due to high computational complexity, as well as difficulties in modeling off-axis wave propagation while accounting for off-axis aberrations. In particular, the common approach of placing the encoding element into the aperture or pupil plane results in only a global control of the wavefront. To overcome these limitations, this work explores an additional design choice by positioning a DOE off-aperture, enabling a spatial unmixing of the degrees of freedom and providing local control over the wavefront over the image plane. Our approach further leverages hybrid refractive-diffractive optical systems by linking differentiable ray and wave optics modeling, thereby optimizing depth imaging quality and demonstrating system versatility. Experimental results reveal that the off-aperture DOE enhances the imaging quality by over 5 dB in PSNR at a FoV of approximately $45^circ$ when paired with a simple thin lens, outperforming traditional on-aperture systems. Furthermore, we successfully recover color and depth information at nearly $28^circ$ FoV using off-aperture DOE configurations with compound optics. Physical prototypes for both applications validate the effectiveness and versatility of the proposed method.

Problem

• Enhancing wide FoV RGBD imaging fidelity
• Overcoming off-axis aberrations in E2E systems
• Optimizing depth imaging with hybrid optics

Innovation

• Off-aperture DOE for local wavefront control
• Hybrid refractive-diffractive optical systems
• Differentiable ray and wave optics modeling