Noisy Spiking Actor Network for Exploration

2024-03-07

arXiv.org

AI Summary

To address the exploration difficulty and excessive noise robustness arising from binary spiking mechanisms in Spiking Neural Networks (SNNs) for deep reinforcement learning, this paper proposes the Temporally Correlated Noise Spiking Action Network (TCN-SAN). The method introduces parameterized, time-domain noise into both the membrane potential accumulation and spike transmission processes of Leaky Integrate-and-Fire (LIF) neurons—marking the first such integration in SNN-based RL. A dynamically decaying noise schedule is further designed to jointly regularize policy stability, thereby balancing exploration efficiency and convergence stability. Evaluated on multiple continuous-control benchmarks in OpenAI Gym, TCN-SAN significantly outperforms existing SNN-based and mainstream deep RL exploration methods, achieving an average performance gain of 12.6% and accelerating policy convergence by 37%.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Learning to Search
• Machine Learning: Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

As a general method for exploration in deep reinforcement learning (RL), NoisyNet can produce problem-specific exploration strategies. Spiking neural networks (SNNs), due to their binary firing mechanism, have strong robustness to noise, making it difficult to realize efficient exploration with local disturbances. To solve this exploration problem, we propose a noisy spiking actor network (NoisySAN) that introduces time-correlated noise during charging and transmission. Moreover, a noise reduction method is proposed to find a stable policy for the agent. Extensive experimental results demonstrate that our method outperforms the state-of-the-art performance on a wide range of continuous control tasks from OpenAI gym.

Problem

• Explores efficient exploration in spiking neural networks for reinforcement learning
• Introduces time-correlated noise to overcome robustness to local disturbances
• Proposes a noise reduction method to achieve stable policy performance

Innovation

• Introduces time-correlated noise in spiking networks
• Proposes noise reduction for stable policy learning
• Outperforms state-of-the-art on continuous control tasks