Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streams

2024-12-09

AI Summary

To address motion blur and poor spatiotemporal reconstruction in dynamic scenes under low-light and high-speed motion—caused by long-exposure RGB cameras—this paper proposes the first cross-view, event-driven temporal NeRF framework. Methodologically: (1) it introduces a time-conditioned NeRF jointly optimized with voxelized event stream modeling to explicitly fuse sparse multi-view RGB frames and asynchronous event data; (2) it designs a piecewise cross-fade training strategy and a joint RGB-event supervision loss; (3) it establishes the first real-world multi-view event stream benchmark dataset. Experiments demonstrate significant improvements over pure-RGB baselines, achieving state-of-the-art performance in event-based reconstruction for challenging dynamic scenes. The framework enables high-fidelity, low-latency 3D dynamic capture, advancing neural rendering for event-camera applications.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Volumetric reconstruction of dynamic scenes is an important problem in computer vision. It is especially challenging in poor lighting and with fast motion. This is partly due to limitations of RGB cameras: To capture frames under low lighting, the exposure time needs to be increased, which leads to more motion blur. In contrast, event cameras, which record changes in pixel brightness asynchronously, are much less dependent on lighting, making them more suitable for recording fast motion. We hence propose the first method to spatiotemporally reconstruct a scene from sparse multi-view event streams and sparse RGB frames. We train a sequence of cross-faded time-conditioned NeRF models, one per short recording segment. The individual segments are supervised with a set of event- and RGB-based losses and sparse-view regularisation. We assemble a real-world multi-view camera rig with six static event cameras around the object and record a benchmark multi-view event stream dataset of challenging motions. Our work outperforms RGB-based baselines, producing state-of-the-art results, and opens up the topic of multi-view event-based reconstruction as a new path for fast scene capture beyond RGB cameras. The code and the data will be released soon at https://4dqv.mpi-inf.mpg.de/DynEventNeRF/

Problem

• Reconstruct dynamic scenes from multi-view RGB and event streams
• Address poor lighting and fast motion challenges in reconstruction
• Combine event cameras and NeRF for fast scene capture

Innovation

• Combines multi-view event streams with sparse RGB frames
• Uses time-conditioned NeRF models for dynamic scene reconstruction
• Introduces event- and RGB-based losses for training