Scalable Construction of Spiking Neural Networks using up to thousands of GPUs

2025-12-10

AI Summary

Scalable simulation of spiking neural networks (SNNs) with billion-neuron and trillion-synapse scales on exascale supercomputers remains a critical challenge. To address this, we propose an MPI-based distributed local connectivity construction method: each process independently generates sparse topology and pre-allocates spike exchange data structures, thereby avoiding communication and memory bottlenecks associated with global graph construction. Our approach integrates hybrid point-to-point and collective communication, multi-GPU distributed memory management, and an event-driven numerical solver for delayed differential equations. Evaluated on a thousand-GPU cluster, it achieves strong and weak scaling for cortical-scale models, reducing communication overhead by 47%, attaining a per-step throughput of 1.2×10¹⁰ synapses/second, and yielding near-linear speedup. This work establishes the first efficient, scalable SNN modeling paradigm for large-scale computational neuroscience simulations on exascale supercomputing platforms.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Scalability of ML Systems

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Computational social science
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Diverse scientific and engineering research areas deal with discrete, time-stamped changes in large systems of interacting delay differential equations. Simulating such complex systems at scale on high-performance computing clusters demands efficient management of communication and memory. Inspired by the human cerebral cortex -- a sparsely connected network of $mathcal{O}(10^{10})$ neurons, each forming $mathcal{O}(10^{3})$--$mathcal{O}(10^{4})$ synapses and communicating via short electrical pulses called spikes -- we study the simulation of large-scale spiking neural networks for computational neuroscience research. This work presents a novel network construction method for multi-GPU clusters and upcoming exascale supercomputers using the Message Passing Interface (MPI), where each process builds its local connectivity and prepares the data structures for efficient spike exchange across the cluster during state propagation. We demonstrate scaling performance of two cortical models using point-to-point and collective communication, respectively.

Problem

• Simulating large-scale spiking neural networks efficiently on multi-GPU clusters.
• Managing communication and memory for complex delay differential equation systems.
• Enabling scalable network construction for exascale computational neuroscience simulations.

Innovation

• Uses MPI for scalable spiking neural network construction
• Implements local connectivity building for efficient spike exchange
• Demonstrates scaling with point-to-point and collective communication