🤖 AI Summary
This study systematically examines the four-decade evolution of synchronization mechanisms and in-network computing architectures in large-scale parallel systems. Tracing the trajectory from the NYU Ultracomputer to modern exascale supercomputers, it integrates key technological milestones—including Fetch-and-Add, multistage interconnection networks, MPI, PCIe atomics, GPU cache coherence mappings, and HIP/Triton compilation stacks—to uncover, for the first time, the dynamic interplay and competition among shared-memory, message-passing, and in-network computing paradigms. The work elucidates the continuous co-evolution of synchronization primitives across hardware-software boundaries, offering critical historical context and architectural insights for the design of future heterogeneous supercomputing systems.
📝 Abstract
This paper presents a historical and technical survey of the hardware architectures, interconnection networks, and synchronization primitives that have shaped massively parallel systems over the past four decades. We examine the design of the NYU Ultracomputer and the IBM Research Parallel Processor Prototype (RP3), focusing on the hardware implementation of the Fetch-and-Add primitive in multistage interconnection networks. We contrast these early attempts at fine-grained, shared-memory hardware combining with the distributed-memory architectures of the IBM SP series and the modern in-network computation models found in NVIDIA SHARP and HPE Slingshot.
We provide a technical analysis of message-passing synchronization, presenting a complete profiling of MPI operation frequencies and detailing the low-level hardware mapping of one-sided RMA atomics to PCIe Atomics and GPU caches. We investigate the software-hardware boundary in modern deep learning, detailing how HIP translation, Triton compilation, and 4-bit quantization (W4A16) execute on modern heterogeneous silicon.
To evaluate alternative network node designs, we present a historical hardware case study analyzing the feasibility of implementing active combining switches using message-passing Inmos Transputers programmed in Occam. Finally, we contextualize the evolution of concurrent software synchronization by examining Isaac Dimitrovsky's parallel "group lock" primitive, tracing its downstream echoes in group mutual exclusion (GME) and room synchronization, and reflect on the historical, philosophical divide between American systems engineering and European formal methods.