Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons

2025-06-03

AI Summary

To address the severe performance degradation of ternary spiking neurons in deep Q-learning, this work identifies gradient estimation bias as the root cause and proposes a low-bias ternary spiking neuron model. The model enhances temporal decision-making representation and training stability by refining both the spike-generation mechanism and the gradient approximation strategy. Integrated with spike-based backpropagation and the Deep Spiking Q-Network (DSQN) architecture, it is systematically evaluated on seven Atari Gym games. Compared to binary baselines, the proposed method achieves a 23.6% average score improvement and fully mitigates the catastrophic performance drop previously observed in ternary spiking models. Experimental results demonstrate its feasibility and superiority for deploying reinforcement learning policies in resource-constrained edge intelligence scenarios.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We propose a new ternary spiking neuron model to improve the representation capacity of binary spiking neurons in deep Q-learning. Although a ternary neuron model has recently been introduced to overcome the limited representation capacity offered by the binary spiking neurons, we show that its performance is worse than that of binary models in deep Q-learning tasks. We hypothesize gradient estimation bias during the training process as the underlying potential cause through mathematical and empirical analysis. We propose a novel ternary spiking neuron model to mitigate this issue by reducing the estimation bias. We use the proposed ternary spiking neuron as the fundamental computing unit in a deep spiking Q-learning network (DSQN) and evaluate the network's performance in seven Atari games from the Gym environment. Results show that the proposed ternary spiking neuron mitigates the drastic performance degradation of ternary neurons in Q-learning tasks and improves the network performance compared to the existing binary neurons, making DSQN a more practical solution for on-board autonomous decision-making tasks.

Problem

• Enhancing representation capacity of binary spiking neurons in deep Q-learning
• Addressing gradient estimation bias in ternary spiking neuron training
• Improving performance of deep spiking Q-learning networks in Atari games

Innovation

• Ternary spiking neuron model reduces estimation bias
• Deep spiking Q-learning network (DSQN) enhances performance
• Improved representation capacity over binary spiking neurons