RGB-Event ISP: The Dataset and Benchmark

2025-01-31

AI Summary

Existing ISP methods underutilize event camera data, particularly failing to address low-light conditions and motion blur in RGB image reconstruction. To bridge this gap, we propose the first event-guided ISP paradigm. We introduce the first pixel-aligned, high-resolution event–RAW paired dataset—comprising 3,373 samples across 24 scenes, 3 exposure levels, and 3 lenses—and establish a standardized benchmarking framework. We categorize learnable ISP methods into three types and design a lightweight event-guided ISP baseline that integrates event features into the classical ISP pipeline (demosaicing, white balancing, denoising, and color transformation). Extensive experiments demonstrate substantial improvements in RGB reconstruction quality under low-light and motion-blurred conditions. Our code and dataset are publicly released to foster deeper integration of event-driven imaging and ISP research.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Event-guided imaging has received significant attention due to its potential to revolutionize instant imaging systems. However, the prior methods primarily focus on enhancing RGB images in a post-processing manner, neglecting the challenges of image signal processor (ISP) dealing with event sensor and the benefits events provide for reforming the ISP process. To achieve this, we conduct the first research on event-guided ISP. First, we present a new event-RAW paired dataset, collected with a novel but still confidential sensor that records pixel-level aligned events and RAW images. This dataset includes 3373 RAW images with 2248 x 3264 resolution and their corresponding events, spanning 24 scenes with 3 exposure modes and 3 lenses. Second, we propose a conventional ISP pipeline to generate good RGB frames as reference. This conventional ISP pipleline performs basic ISP operations, e.g.demosaicing, white balancing, denoising and color space transforming, with a ColorChecker as reference. Third, we classify the existing learnable ISP methods into 3 classes, and select multiple methods to train and evaluate on our new dataset. Lastly, since there is no prior work for reference, we propose a simple event-guided ISP method and test it on our dataset. We further put forward key technical challenges and future directions in RGB-Event ISP. In summary, to the best of our knowledge, this is the very first research focusing on event-guided ISP, and we hope it will inspire the community. The code and dataset are available at: https://github.com/yunfanLu/RGB-Event-ISP.

Problem

• Event Sensors
• Image Signal Processing
• RGB Image Processing

Innovation

• RGB-Event ISP
• Event-guided Imaging
• ISP Algorithm Optimization