Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging

2024-04-17

IEEE Transactions on Pattern Analysis and Machine Intelligence

AI Summary

Traditional cameras face a fundamental noise-blur trade-off in low-light, high-speed imaging: long exposures induce motion blur, while short exposures severely degrade signal-to-noise ratio (SNR). This work introduces the first complementary fusion framework integrating event cameras and single-photon avalanche diode (SPAD) arrays. It leverages the event camera’s asynchronous, microsecond-resolution (up to 1 MHz) detection of brightness changes and the SPAD array’s single-photon sensitivity with picosecond-precision temporal stamping. A joint spatiotemporal reconstruction algorithm is proposed, validated on both synthetic and real-world datasets. Experiments demonstrate that, at 100 kHz temporal resolution, the method achieves over 5 dB PSNR improvement in low-light reconstruction compared to conventional approaches—simultaneously preserving motion sharpness and SNR. This synergy establishes a new paradigm for high-speed, low-light vision, with direct implications for robotic perception and real-time biomedical imaging.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Traditional cameras face a trade-off between low-light performance and high-speed imaging: longer exposure times to capture sufficient light results in motion blur, whereas shorter exposures result in Poisson-corrupted noisy images. While burst photography techniques help mitigate this tradeoff, conventional cameras are fundamentally limited in their sensor noise characteristics. Event cameras and single-photon avalanche diode (SPAD) sensors have emerged as promising alternatives to conventional cameras due to their desirable properties. SPADs are capable of single-photon sensitivity with microsecond temporal resolution, and event cameras can measure brightness changes up to 1 MHz with low bandwidth requirements. We show that these properties are complementary, and can help achieve low-light, high-speed image reconstruction with low bandwidth requirements. We introduce a sensor fusion framework to combine SPADs with event cameras to improves the reconstruction of high-speed, low-light scenes while reducing the high bandwidth cost associated with using every SPAD frame. Our evaluation, on both synthetic and real sensor data, demonstrates significant enhancements ($gt 5$ dB PSNR) in reconstructing low-light scenes at high temporal resolution (100 kHz) compared to conventional cameras. Event-SPAD fusion shows great promise for real-world applications, such as robotics or medical imaging.

Problem

• Overcoming trade-off between low-light and high-speed imaging
• Reducing bandwidth cost in high-speed image reconstruction
• Enhancing low-light scene reconstruction at high temporal resolution

Innovation

• Combines SPADs and event cameras
• Enables high-speed low-light imaging
• Reduces bandwidth with sensor fusion