Hardware-aware Coding Function Design for Compressive Single-Photon 3D Cameras

2025-10-13

AI Summary

Single-photon time-of-flight (ToF) 3D imaging is fundamentally limited by system bandwidth, laser peak power, and sensor memory and computational capacity. Conventional compressed histogram approaches reduce data rates but fail to respect realistic hardware constraints. This paper proposes a hardware-aware joint optimization framework for structured illumination patterns and sensing matrices within a compressed sensing paradigm. Crucially, bandwidth, peak optical power, and sensor resource limitations are explicitly formulated as differentiable constraints. The entire pipeline is trained end-to-end via gradient descent, enabling adaptive compensation for non-ideal impulse responses. Experimental results—both in simulation and on physical hardware—demonstrate substantial improvements over conventional coding schemes: under peak-power constraints, depth reconstruction error is reduced by up to 32%. To the best of our knowledge, this work represents the first approach to learn hardware-constrained coding patterns while ensuring robust 3D reconstruction.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Constraint Satisfaction and Optimization: Constraint Optimization
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Single-photon cameras are becoming increasingly popular in time-of-flight 3D imaging because they can time-tag individual photons with extreme resolution. However, their performance is susceptible to hardware limitations, such as system bandwidth, maximum laser power, sensor data rates, and in-sensor memory and compute resources. Compressive histograms were recently introduced as a solution to the challenge of data rates through an online in-sensor compression of photon timestamp data. Although compressive histograms work within limited in-sensor memory and computational resources, they underperform when subjected to real-world illumination hardware constraints. To address this, we present a constrained optimization approach for designing practical coding functions for compressive single-photon 3D imaging. Using gradient descent, we jointly optimize an illumination and coding matrix (i.e., the coding functions) that adheres to hardware constraints. We show through extensive simulations that our coding functions consistently outperform traditional coding designs under both bandwidth and peak power constraints. This advantage is particularly pronounced in systems constrained by peak power. Finally, we show that our approach adapts to arbitrary parameterized impulse responses by evaluating it on a real-world system with a non-ideal impulse response function.

Problem

• Designs coding functions for compressive single-photon 3D cameras under hardware constraints
• Optimizes illumination and coding matrices using gradient descent for bandwidth limitations
• Addresses peak power constraints and non-ideal impulse responses in real systems

Innovation

• Constrained optimization for coding function design
• Gradient descent to jointly optimize illumination and coding
• Adapts to arbitrary parameterized impulse responses