Machine vision with small numbers of detected photons per inference

2026-03-25

AI Summary

This work addresses the fundamental challenge of visual perception under extremely low-light conditions, where conventional imaging systems fail due to severe photon scarcity (average photons per pixel ≤ 1). The authors propose Photon-Aware Neuromorphic Sensing (PANS), a novel framework that explicitly models the stochastic nature of photon detection and integrates it into an end-to-end co-design of optics and algorithms. PANS achieves unprecedented recognition performance at the physical limit of just a few to tens of photons per frame, attaining 73% and 86% classification accuracy on FashionMNIST and MNIST with only 4.9 and 8.6 photons per image on average, respectively. This represents a photon efficiency improvement of several orders of magnitude over traditional approaches, while also demonstrating strong generalization capabilities in event detection and image reconstruction tasks.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Hardware-aware ML

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Machine vision, including object recognition and image reconstruction, is a central technology in many consumer devices and scientific instruments. The design of machine-vision systems has been revolutionized by the adoption of end-to-end optimization, in which the optical front end and the post-processing back end are jointly optimized. However, while machine vision currently works extremely well in moderate-light or bright-light situations -- where a camera may detect thousands of photons per pixel and billions of photons per frame -- it is far more challenging in very low-light situations. We introduce photon-aware neuromorphic sensing (PANS), an approach for end-to-end optimization in highly photon-starved scenarios. The training incorporates knowledge of the low photon budget and the stochastic nature of light detection when the average number of photons per pixel is near or less than 1. We report a proof-of-principle experimental demonstration in which we performed low-light image classification using PANS, achieving 73% (82%) accuracy on FashionMNIST with an average of only 4.9 (17) detected photons in total per inference, and 86% (97%) on MNIST with 8.6 (29) detected photons -- orders of magnitude more photon-efficient than conventional approaches. We also report simulation studies showing how PANS could be applied to other classification, event-detection, and image-reconstruction tasks. By taking into account the statistics of measurement results for non-classical states or alternative sensing hardware, PANS could in principle be adapted to enable high-accuracy results in quantum and other photon-starved setups.

Problem

• machine vision
• low-light imaging
• photon-starved
• image classification
• stochastic detection

Innovation

• photon-starved imaging
• end-to-end optimization
• neuromorphic sensing
• low-light machine vision
• quantum-inspired sensing