SpikeGS: Reconstruct 3D scene via fast-moving bio-inspired sensors

2024-07-04

arXiv.org

AI Summary

To address the failure of 3D Gaussian Splatting (3DGS) in high-speed camera motion scenarios—caused by motion blur and texture degradation—this paper introduces event streams from neuromorphic (pulse-based) vision sensors into the 3DGS framework, proposing an Event-Gaussian Fusion architecture. Our method leverages event data to enable robust reconstruction under rapid motion. Key contributions include: (1) accumulative rasterization rendering, enabling efficient, differentiable mapping from sparse event streams to intensity images; (2) interval-wise supervision, mitigating mismatches between event sparsity and continuous motion modeling; and (3) biologically inspired motion modeling jointly optimized with 3D Gaussian parameters. Evaluated on both synthetic and real-world dynamic scenes, our approach achieves high-fidelity joint geometry and texture reconstruction within one second. Quantitatively, it improves PSNR by 3.2 dB over state-of-the-art event-driven and deblurring methods, significantly enhancing the practicality of 3DGS for fast-motion scenarios.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

3D Gaussian Splatting (3DGS) demonstrates unparalleled superior performance in 3D scene reconstruction. However, 3DGS heavily relies on the sharp images. Fulfilling this requirement can be challenging in real-world scenarios especially when the camera moves fast, which severely limits the application of 3DGS. To address these challenges, we proposed Spike Gausian Splatting (SpikeGS), the first framework that integrates the spike streams into 3DGS pipeline to reconstruct 3D scenes via a fast-moving bio-inspired camera. With accumulation rasterization, interval supervision, and a specially designed pipeline, SpikeGS extracts detailed geometry and texture from high temporal resolution but texture lacking spike stream, reconstructs 3D scenes captured in 1 second. Extensive experiments on multiple synthetic and real-world datasets demonstrate the superiority of SpikeGS compared with existing spike-based and deblur 3D scene reconstruction methods. Codes and data will be released soon.

Problem

• 3D Gaussian Splats
• High-speed Camera Movement
• Image Clarity

Innovation

• SpikeGS
• 3D Gaussian Splatting
• Texture-poor Conditions