Ultra-low-light computer vision using trained photon correlations

2026-04-13

AI Summary

This work addresses the challenge of accurate object recognition under extremely low-light and high-noise conditions, where conventional vision systems typically fail. The authors propose a hybrid optoelectronic vision system that, for the first time, enables end-to-end joint optimization of a programmable correlated photon illumination source with a Transformer-based backend. Central to this approach is Correlation-Aware Training (CAT), a novel strategy explicitly designed for object recognition rather than image reconstruction. By exploiting spatial correlations among photons, the system achieves substantial performance gains under an ultra-low photon budget of ≤100 exposures. Experimental results demonstrate that the proposed method improves object recognition accuracy by up to 15 percentage points compared to both traditional uncorrelated illumination and untrained correlated schemes, thereby overcoming a key performance bottleneck in low-light imaging.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Learning to Search
• Machine Learning: Hardware-aware ML

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Illumination using correlated photon sources has been established as an approach to allowing high-fidelity images to be reconstructed from noisy camera frames by taking advantage of the knowledge that signal photons are spatially correlated whereas detector clicks due to noise are uncorrelated. However, in computer-vision tasks, the goal is often not ultimately to reconstruct an image, but to make inferences about a scene -- such as what object is present. Here we show how correlated-photon illumination can be used to gain an advantage in a hybrid optical-electronic computer-vision pipeline for object recognition. We demonstrate correlation-aware training (CAT): end-to-end optimization of a trainable correlated-photon illumination source and a Transformer backend in a way that the Transformer can learn to benefit from the correlations, using a small number (<= 100) of shots. We show a classification accuracy enhancement of up to 15 percentage points over conventional, uncorrelated-illumination-based computer vision in ultra-low-light and noisy imaging conditions, as well as an improvement over using untrained correlated-photon illumination. Our work illustrates how specializing to a computer-vision task -- object recognition -- and training the pattern of photon correlations in conjunction with a digital backend allows us to push the limits of accuracy in highly photon-budget-constrained scenarios beyond existing methods focused on image reconstruction.

Problem

• ultra-low-light
• computer vision
• object recognition
• photon correlations
• noisy imaging

Innovation

• correlated-photon illumination
• correlation-aware training
• ultra-low-light computer vision
• end-to-end optimization
• photon-efficient imaging