E-RGB-D: Real-Time Event-Based Perception with Structured Light

2025-12-20

AI Summary

Conventional monochrome event cameras suffer from poor sensitivity to static or low-velocity objects and lack color and depth information. To address these limitations, this work proposes a real-time RGB-D perception framework integrating an event camera with dynamic structured light. We introduce the first event-driven, per-pixel independent color-depth synchronization encoding mechanism. The system employs a TI LightCrafter 4500 DLP projector and a monocular event camera, leveraging active structured-light coding, hardware-level event–light synchronization triggering, and a real-time point-cloud coloring algorithm. While maintaining low bandwidth, it achieves equivalent 1400 fps color object detection and 4 kHz per-pixel depth output, generating high-fidelity colored point clouds. This is the first work to realize pixel-level joint RGB-D perception on event cameras. The source code is publicly available.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Hardware-aware ML

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Event-based cameras (ECs) have emerged as bio-inspired sensors that report pixel brightness changes asynchronously, offering unmatched speed and efficiency in vision sensing. Despite their high dynamic range, temporal resolution, low power consumption, and computational simplicity, traditional monochrome ECs face limitations in detecting static or slowly moving objects and lack color information essential for certain applications. To address these challenges, we present a novel approach that integrates a Digital Light Processing (DLP) projector, forming Active Structured Light (ASL) for RGB-D sensing. By combining the benefits of ECs and projection-based techniques, our method enables the detection of color and the depth of each pixel separately. Dynamic projection adjustments optimize bandwidth, ensuring selective color data acquisition and yielding colorful point clouds without sacrificing spatial resolution. This integration, facilitated by a commercial TI LightCrafter 4500 projector and a monocular monochrome EC, not only enables frameless RGB-D sensing applications but also achieves remarkable performance milestones. With our approach, we achieved a color detection speed equivalent to 1400 fps and 4 kHz of pixel depth detection, significantly advancing the realm of computer vision across diverse fields from robotics to 3D reconstruction methods. Our code is publicly available: https://github.com/MISTLab/event_based_rgbd_ros

Problem

• Enables color and depth detection for event-based cameras
• Overcomes static object detection limitations in monochrome event cameras
• Achieves high-speed RGB-D sensing with dynamic projection optimization

Innovation

• Integrates DLP projector for Active Structured Light RGB-D sensing
• Enables separate color and depth detection per pixel
• Achieves 1400 fps color and 4 kHz depth detection speeds