SNNSIR: A Simple Spiking Neural Network for Stereo Image Restoration

2025-08-17

AI Summary

To address the challenge of balancing energy efficiency and performance in stereo image restoration using Spiking Neural Networks (SNNs), this paper proposes the first fully spike-driven lightweight stereo restoration framework. Methodologically, we introduce a spike-based residual basic block, spike stereo convolutional modulation, and a spike stereo cross-attention module—eliminating all floating-point matrix operations in favor of event-driven binary spike computation, element-wise multiplicative modulation, and cross-view spike attention. Our contributions are threefold: (1) the first pure-spike paradigm enabling multi-task stereo restoration—including deraining, dehazing, low-light enhancement, and super-resolution; (2) substantial reductions in computational cost and energy consumption, enabling hardware-friendly deployment; and (3) state-of-the-art performance across multiple stereo restoration tasks, demonstrating feasibility and superiority for real-time, low-power stereo vision systems.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Spiking Neural Networks (SNNs), characterized by discrete binary activations, offer high computational efficiency and low energy consumption, making them well-suited for computation-intensive tasks such as stereo image restoration. In this work, we propose SNNSIR, a simple yet effective Spiking Neural Network for Stereo Image Restoration, specifically designed under the spike-driven paradigm where neurons transmit information through sparse, event-based binary spikes. In contrast to existing hybrid SNN-ANN models that still rely on operations such as floating-point matrix division or exponentiation, which are incompatible with the binary and event-driven nature of SNNs, our proposed SNNSIR adopts a fully spike-driven architecture to achieve low-power and hardware-friendly computation. To address the expressiveness limitations of binary spiking neurons, we first introduce a lightweight Spike Residual Basic Block (SRBB) to enhance information flow via spike-compatible residual learning. Building on this, the Spike Stereo Convolutional Modulation (SSCM) module introduces simplified nonlinearity through element-wise multiplication and highlights noise-sensitive regions via cross-view-aware modulation. Complementing this, the Spike Stereo Cross-Attention (SSCA) module further improves stereo correspondence by enabling efficient bidirectional feature interaction across views within a spike-compatible framework. Extensive experiments on diverse stereo image restoration tasks, including rain streak removal, raindrop removal, low-light enhancement, and super-resolution demonstrate that our model achieves competitive restoration performance while significantly reducing computational overhead. These results highlight the potential for real-time, low-power stereo vision applications. The code will be available after the article is accepted.

Problem

• Develops SNNSIR for efficient stereo image restoration
• Addresses binary spike limitations with spike-compatible modules
• Reduces computational overhead in stereo vision tasks

Innovation

• Fully spike-driven architecture for efficiency
• Spike Residual Basic Block enhances information flow
• Spike Stereo Convolutional Modulation simplifies nonlinearity