UltraLIF: Fully Differentiable Spiking Neural Networks via Ultradiscretization and Max-Plus Algebra

2026-02-10

AI Summary

This work addresses the challenge of training spiking neural networks (SNNs), which typically rely on heuristic surrogate gradients due to the non-differentiability of spikes, leading to training instability and forward–backward inconsistency. For the first time, the authors introduce hyper-discretization from tropical geometry into SNNs, constructing a differentiable continuous relaxation model via max-plus algebra and a log-sum-exp soft thresholding operator. They rigorously establish its theoretical convergence to the Leaky Integrate-and-Fire (LIF) neuron dynamics, thereby eliminating the need for surrogate gradients and enabling standard backpropagation with spatiotemporal modeling. Experiments demonstrate consistent superiority over existing surrogate gradient methods across six benchmarks—including static images, neuromorphic vision, and audio tasks—with notable performance gains under single-timestep (T=1) settings and significant energy savings achievable through sparsity-inducing penalties.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

Spiking Neural Networks (SNNs) offer energy-efficient, biologically plausible computation but suffer from non-differentiable spike generation, necessitating reliance on heuristic surrogate gradients. This paper introduces UltraLIF, a principled framework that replaces surrogate gradients with ultradiscretization, a mathematical formalism from tropical geometry providing continuous relaxations of discrete dynamics. The central insight is that the max-plus semiring underlying ultradiscretization naturally models neural threshold dynamics: the log-sum-exp function serves as a differentiable soft-maximum that converges to hard thresholding as a learnable temperature parameter $\eps \to 0$. Two neuron models are derived from distinct dynamical systems: UltraLIF from the LIF ordinary differential equation (temporal dynamics) and UltraDLIF from the diffusion equation modeling gap junction coupling across neuronal populations (spatial dynamics). Both yield fully differentiable SNNs trainable via standard backpropagation with no forward-backward mismatch. Theoretical analysis establishes pointwise convergence to classical LIF dynamics with quantitative error bounds and bounded non-vanishing gradients. Experiments on six benchmarks spanning static images, neuromorphic vision, and audio demonstrate improvements over surrogate gradient baselines, with gains most pronounced in single-timestep ($T{=}1$) settings on neuromorphic and temporal datasets. An optional sparsity penalty enables significant energy reduction while maintaining competitive accuracy.

Problem

• Spiking Neural Networks
• non-differentiable spike generation
• surrogate gradients
• forward-backward mismatch
• ultradiscretization

Innovation

• ultradiscretization
• max-plus algebra
• fully differentiable SNNs
• log-sum-exp relaxation
• spiking neural networks