Efficient 3D Recognition with Event-driven Spike Sparse Convolution

2024-12-10

arXiv.org

AI Summary

To address performance bottlenecks in spiking neural networks (SNNs) arising from mismatches between preprocessing and feature extraction for sparse 3D point clouds, this paper proposes E-3DSNN—the first end-to-end trainable 3D SNN backbone supporting classification, detection, and segmentation jointly. Its core contributions are: (1) Spike Voxel Coding, a novel sparse voxel-based spike encoding scheme tailored for event-driven 3D data; and (2) Spike Sparse Convolution, integrating event-driven convolution with brain-inspired, hardware-efficient address mapping. Evaluated on ModelNet40, E-3DSNN achieves 91.7% accuracy—surpassing the state-of-the-art by 3.0 percentage points—with only 1.87M parameters. On KITTI benchmarks, it attains the highest energy efficiency to date, enabling real-time inference on neuromorphic hardware.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: 3D Computer Vision
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Spiking Neural Networks (SNNs) provide an energy-efficient way to extract 3D spatio-temporal features. Point clouds are sparse 3D spatial data, which suggests that SNNs should be well-suited for processing them. However, when applying SNNs to point clouds, they often exhibit limited performance and fewer application scenarios. We attribute this to inappropriate preprocessing and feature extraction methods. To address this issue, we first introduce the Spike Voxel Coding (SVC) scheme, which encodes the 3D point clouds into a sparse spike train space, reducing the storage requirements and saving time on point cloud preprocessing. Then, we propose a Spike Sparse Convolution (SSC) model for efficiently extracting 3D sparse point cloud features. Combining SVC and SSC, we design an efficient 3D SNN backbone (E-3DSNN), which is friendly with neuromorphic hardware. For instance, SSC can be implemented on neuromorphic chips with only minor modifications to the addressing function of vanilla spike convolution. Experiments on ModelNet40, KITTI, and Semantic KITTI datasets demonstrate that E-3DSNN achieves state-of-the-art (SOTA) results with remarkable efficiency. Notably, our E-3DSNN (1.87M) obtained 91.7% top-1 accuracy on ModelNet40, surpassing the current best SNN baselines (14.3M) by 3.0%. To our best knowledge, it is the first direct training 3D SNN backbone that can simultaneously handle various 3D computer vision tasks (e.g., classification, detection, and segmentation) with an event-driven nature. Code is available: https://github.com/bollossom/E-3DSNN/.

Problem

• Enhance 3D recognition using Spiking Neural Networks.
• Improve feature extraction from sparse 3D point clouds.
• Optimize SNN performance for diverse 3D vision tasks.

Innovation

• Introduces Spike Voxel Coding (SVC)
• Proposes Spike Sparse Convolution (SSC)
• Designs efficient 3D SNN backbone (E-3DSNN)