Hybrid Event Frame Sensors: Modeling, Calibration, and Simulation

2025-11-22

AI Summary

Hybrid event-frame sensors (APS+EVS) suffer from complex, circuit-coupled noise patterns that are difficult to model, and existing methods lack a unified characterization of noise interactions between APS and EVS pixels. To address this, we propose the first joint statistical noise model, which quantitatively reveals the dependence of EVS noise on illumination intensity and dark current—previously uncharacterized. We design an end-to-end joint calibration pipeline that simultaneously estimates photon shot noise, dark current noise, fixed-pattern noise, and quantization noise for both APS and EVS pixels. Based on this model, we develop HESIM, a high-fidelity simulator capable of co-generating realistic RAW frames and event streams. Extensive validation on two real-world hybrid sensors demonstrates that model parameters are reliably identifiable from empirical measurements, and synthetic data generated by HESIM significantly improves downstream task performance—including video frame interpolation and motion deblurring—upon domain transfer.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Event frame hybrid sensors integrate an Active Pixel Sensor (APS) and an Event Vision Sensor (EVS) within a single chip, combining the high dynamic range and low latency of the EVS with the rich spatial intensity information from the APS. While this tight integration offers compact, temporally precise imaging, the complex circuit architecture introduces non-trivial noise patterns that remain poorly understood and unmodeled. In this work, we present the first unified, statistics-based imaging noise model that jointly describes the noise behavior of APS and EVS pixels. Our formulation explicitly incorporates photon shot noise, dark current noise, fixed-pattern noise, and quantization noise, and links EVS noise to illumination level and dark current. Based on this formulation, we further develop a calibration pipeline to estimate noise parameters from real data and offer a detailed analysis of both APS and EVS noise behaviors. Finally, we propose HESIM, a statistically grounded simulator that generates RAW frames and events under realistic, jointly calibrated noise statistics. Experiments on two hybrid sensors validate our model across multiple imaging tasks (e.g., video frame interpolation and deblurring), demonstrating strong transfer from simulation to real data.

Problem

• Modeling complex noise patterns in hybrid event frame sensors
• Developing unified noise model for APS and EVS pixel behaviors
• Creating calibrated simulator for realistic sensor noise generation

Innovation

• Unified noise model for APS and EVS sensors
• Calibration pipeline estimating noise parameters from data
• HESIM simulator generating frames with realistic noise