SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers

2026-08-13

AI Summary

This study addresses the suboptimal training caused by fixed surrogate gradients in spiking Transformers by proposing the SAGE mechanism. SAGE quantifies uncertainty via normalized self-attention entropy to adaptively modulate surrogate gradient slopes, enabling dynamic parameter optimization with zero inference overhead. By preserving the inference model unchanged, this approach significantly enhances optimization flexibility. Experiments on CIFAR-10/100 demonstrate that SAGE consistently improves accuracy by 1–2% over fixed baselines, effectively resolving the trade-off between gradient estimation and model performance in spiking neural networks.

Abstract

Spiking neural networks (SNNs) offer an energy-efficient alternative to conventional deep neural networks by exploiting sparse event-driven computation, but their training remains challenging because the non-differentiable spike function requires surrogate gradients whose fixed shape may be suboptimal across layers and training stages. In this work, we introduce SAGE, an uncertainty-modulated surrogate-gradient mechanism for Transformer-based SNNs. SAGE estimates block-level uncertainty from normalized self-attention entropy and uses this signal to adapt the surrogate-gradient slope during training while leaving the inference model unchanged. By modulating only the training-time surrogate parameter, the proposed method preserves the original architecture and deployment cost while improving optimization flexibility. Experiments on CIFAR-10/100 demonstrate that SAGE achieves improved accuracy over fixed-surrogate baselines, with results up to 1-2\% consistent gains across multiple simulation time steps. These results highlight the potential of attention-derived uncertainty as a lightweight training signal for adaptive surrogate-gradient learning in transformer-based SNNs.

Problem

• Spiking Neural Networks
• Surrogate Gradient
• Spiking Transformers
• Non-differentiable Spike Function
• Training Optimization

Innovation

• Surrogate Gradient Adaptation
• Attention-Guided Entropy
• Spiking Transformers
• Uncertainty Modulation
• Training-time Adaptation