ADMM-Based Training for Spiking Neural Networks

2025-05-08

AI Summary

Spike-based neural networks (SNNs) suffer from training difficulties due to the non-differentiability of step-like activation functions, while existing surrogate gradient methods exhibit numerical instability and poor scalability. To address this, this work introduces the Alternating Direction Method of Multipliers (ADMM) into end-to-end SNN training for the first time. By reformulating the optimization problem and deriving closed-form parameter update rules, our approach explicitly handles the non-smooth activation constraints without relying on gradient approximations. Experiments demonstrate stable convergence, significantly improved training robustness, and superior energy efficiency across multiple temporal tasks—outperforming mainstream SGD-based baselines. This work establishes a gradient-free optimization paradigm for SNNs, enabling low-power, scalable neuromorphic learning.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

In recent years, spiking neural networks (SNNs) have gained momentum due to their high potential in time-series processing combined with minimal energy consumption. However, they still lack a dedicated and efficient training algorithm. The popular backpropagation with surrogate gradients, adapted from stochastic gradient descent (SGD)-derived algorithms, has several drawbacks when used as an optimizer for SNNs. Specifically, it suffers from low scalability and numerical imprecision. In this paper, we propose a novel SNN training method based on the alternating direction method of multipliers (ADMM). Our ADMM-based training aims to solve the problem of the SNN step function's non-differentiability. We formulate the problem, derive closed-form updates, and empirically show the optimizer's convergence properties, great potential, and possible new research directions to improve the method in a simulated proof-of-concept.

Problem

• Lack of efficient training algorithm for SNNs
• Backpropagation suffers from scalability and imprecision
• ADMM-based training addresses non-differentiability of SNN step function

Innovation

• ADMM-based training for SNNs
• Solves non-differentiability of step function
• Closed-form updates for convergence