EsaacSim: A Multimodal Event Camera Add-on for NVIDIA Isaac Sim

2026-08-09

AI Summary

This work addresses the limited adoption of event cameras in robotics due to hardware scarcity and the absence of simulation tools compatible with modern platforms. We present the first multimodal event camera plugin for NVIDIA Isaac Sim, supporting both grayscale and Bayer RGGB modes while synchronously outputting RGB, APS frames, event streams, depth, and IMU data. To enhance effective temporal resolution without compromising compatibility with Isaac Sim’s rendering pipeline, we introduce a motion-guided inter-frame synthesis strategy. Built on native ROS 2 interfaces and supporting multiple resolutions, the plugin incurs less than 400 MB of additional GPU memory on an RTX 4060, enabling real-time simulation at equivalent event rates of 240–960 Hz, with per-frame generation latency as low as 6.98 ms (grayscale) and 7.58 ms (Bayer RGGB).

Abstract

Event-based vision is becoming an increasingly important sensing paradigm for robotics, yet its adoption remains limited by sensor availability and the lack of integrated simulation tools for modern robotics platforms. This paper presents EsaacSim, a multimodal event camera add-on for NVIDIA Isaac Sim that enables online simulation of configurable event cameras with grayscale and Bayer RGGB event generation. The framework supports multiple event camera resolutions and provides synchronized RGB, APS, event, depth, and IMU outputs through native ROS2 interfaces. A motion-guided frame-gap synthesis strategy further increases the effective temporal resolution while preserving compatibility with the Isaac Sim rendering pipeline. Experimental evaluation demonstrates synchronized multimodal simulation across representative robotic scenes and efficient online performance over five event camera resolutions at effective event rates from 240 to 960Hz. Event stream generation requires 6.98--27.28ms for grayscale events and 7.58--29.16ms for Bayer RGGB events while using less than 400MB of additional GPU memory on an NVIDIA RTX~4060 GPU. These results show that EsaacSim enables supports online multimodal event-camera simulation for robotics research and synthetic data generation. We release an early version of the simulator and report its current architecture and performance.

Problem

• event-based vision
• robotics simulation
• multimodal sensing
• event camera
• NVIDIA Isaac Sim

Innovation

• event-based vision
• multimodal simulation
• NVIDIA Isaac Sim
• motion-guided synthesis
• ROS2 integration