parallelization technique design

This study systematically examines the four-decade evolution of synchronization mechanisms and in-network computing architectures in large-scale parallel systems. Tracing the trajectory from the NYU Ultracomputer to modern exascale supercomputers, it integrates key technological milestones—including Fetch-and-Add, multistage interconnection networks, MPI, PCIe atomics, GPU cache coherence mappings, and HIP/Triton compilation stacks—to uncover, for the first time, the dynamic interplay and competition among shared-memory, message-passing, and in-network computing paradigms. The work elucidates the continuous co-evolution of synchronization primitives across hardware-software boundaries, offering critical historical context and architectural insights for the design of future heterogeneous supercomputing systems.

HPX suffers from suboptimal C++ Executor performance on heterogeneous hardware due to static resource allocation. To address this, we propose a cores-aware and chunking-aware adaptive executor model that dynamically monitors runtime load, models scheduling overhead, and heuristically adjusts task chunking and core binding—enabling online optimization for both compute- and memory-bound workloads within HPX. Our design fully conforms to the standard C++20 Executor interface and requires no modifications to user code. Experimental evaluation across diverse hardware configurations and representative parallel workloads demonstrates speedups of 1.4–2.3× over baseline static strategies, confirming substantial performance gains while preserving portability and standards compliance.

Existing parallel computing curricula for undergraduate and graduate students often lack a unified, principle-centered pedagogical framework that balances theoretical foundations with practical implementation while ensuring broad applicability. Method: This work develops a systematic lecture note suite grounded in deterministic parallel algorithms, covering core theory (work-time model, efficiency and scalability analysis), mainstream programming models (OpenMP, MPI, pthreads), and C-language implementation—explicitly excluding GPU programming and randomized algorithms to preserve conceptual generality. It integrates visualization-guided explanations, verifiable code examples, and structured programming exercises emphasizing universal performance criteria: execution time, energy consumption, and scalability. Contribution/Results: The resulting self-contained, production-ready lecture notes are accompanied by open-source code and extensible problem sets. They effectively support both formal instruction in parallel and high-performance computing courses and independent learning, enhancing pedagogical coherence and practical accessibility.

Existing user-space coroutine/fiber synchronization mechanisms implicitly assume kernel scheduling, introducing unnecessary latency on critical paths and limiting high-concurrency throughput. This paper proposes Combine-and-Exchange Scheduling (CES), a novel synchronization paradigm for purely user-space cooperative scheduling. CES eliminates cross-thread overhead by retaining critical sections on the same thread during lock contention, while dynamically redistributing parallelizable tasks to idle threads. Crucially, it co-designs user-space synchronization primitives with the scheduler to fully bypass kernel intervention. Experimental evaluation demonstrates that CES achieves up to 3× higher throughput on application-level benchmarks and up to 8× speedup on microbenchmarks—significantly outperforming state-of-the-art user-space synchronization approaches.

SYCL programs on multi-GPU clusters suffer from high scheduling latency and substantial critical-path overhead due to implicit memory allocation, cache-coherence operations, and dependency analysis. Method: We propose the Instruction Graph—a novel intermediate representation that fully decouples scheduling from execution. Our approach integrates speculative scheduling, adaptive virtual-buffer memory allocation, and tight integration with the Celerity runtime, enabling fully concurrent scheduling of memory management, data transfers, MPI communication, and kernel launches while moving all scheduling analysis off the critical execution path. Contribution/Results: Evaluated on a production-scale 128-GPU cluster, our method achieves excellent strong scaling, drastically reduces multi-application scheduling latency, and drives critical-path overhead nearly to zero—thereby overcoming fundamental limitations of conventional static and blocking schedulers.

This study investigates the scalability and performance of process and thread schedulers under memory-intensive workloads in multi-core shared-memory systems, focusing on a 3D tensor row-sorting task. The authors design and evaluate several scheduling strategies: on the thread side, an AIMD-based adaptive chunking mechanism inspired by TCP congestion control is introduced, coupled with exponential weighted moving average to dynamically adjust concurrency; on the process side, a bounded prolific/collective model is employed alongside one-to-one, one-to-many, and many-to-many pipelined communication patterns to enable flexible task distribution. Experimental results on a 24-core x86-64 platform demonstrate that thread-level scheduling consistently outperforms process-level scheduling, with dynamic and guided strategies achieving the best performance, while the many-to-many pipeline exhibits superior scalability for large-scale tasks.

This work addresses the challenges of latency and scalability in designing efficient concurrent primitives under high write contention in shared-memory systems. It introduces a novel approach based on a contention-resolution algorithm that transforms contention-prone hardware primitives into higher-level concurrent objects within an approximately synchronous randomized scheduling model. For the first time, the study achieves composable, low-latency concurrent primitives against an adaptive adversary, and establishes a theoretical lower bound for the space–latency tradeoff. Using only O(1) read–write registers and a single compare-and-swap (CAS) register, the construction yields—with high probability—O(log P) latency for a variety of primitives, including read–write registers, CAS, load-linked/store-conditional (LL/SC), fetch-and-increment, bounded max registers, and counters.

This work addresses the poor performance and high contention of shared stack access under high concurrency by proposing a sharded, blocking linearizable stack design. The approach innovatively integrates an elimination mechanism with request combining and employs a lightweight fetch-and-increment atomic operation to coordinate access across shards. This strategy effectively reduces thread contention and substantially improves parallel efficiency. Experimental results demonstrate that, in highly contended multi-threaded scenarios, the proposed design outperforms existing concurrent stack implementations across most workloads, achieving up to a 2× speedup.

This work addresses the limitations of traditional numerical array programs, which rely on manual parallelization constrained by static optimizations or explicit annotations, resulting in coarse-grained parallelism and poor adaptability to heterogeneous hardware. The paper proposes a self-optimizing Virtual Processor (VP) that automatically and dynamically parallelizes entire program regions at runtime through a decentralized network of collaborating execution segments, without developer intervention. Its key innovation lies in parallelizing and distributing the scheduling process itself, integrating dependency-driven local decisions, heterogeneity-aware task placement and data movement, and support from the ILNumerics.ONAL instruction set. This approach preserves sequential semantics while enabling automatic parallelism extraction across large-scale program regions, achieving low-latency strong scaling on local heterogeneous systems for a broad range of workloads—from latency-sensitive small operations to large data-parallel tasks—without requiring explicit parallel programming.

This study addresses the limited understanding of the synergistic interplay among FP8 matrix cores, asynchronous execution, and 2:4 structured sparsity on AMD MI300A systems, a gap that hinders performance optimization for HPC and HPC-AI workloads. Through custom microbenchmarks, Transformer-style kernels, and mixed-precision concurrency tests—augmented by hardware performance counters and runtime analysis—the work provides the first systematic characterization of their interaction dynamics. It quantifies concurrency throughput and fairness boundaries, revealing the context-dependent efficacy of structured sparsity. The research identifies critical occupancy thresholds and conditions under which sparsity yields tangible benefits, and proposes practical scheduling and sparsity activation strategies for real-world deployment, offering actionable guidance for performance tuning on MI300A-class architectures.

This work extends ACETONE, originally limited to generating sequential C code, to support multicore parallel code generation, thereby unlocking the parallel performance potential of multicore embedded platforms. By formally modeling the processor allocation problem, designing a scheduling heuristic, and introducing customized synchronization mechanism templates, the approach enables efficient mapping and execution of neural network layers across multicore architectures. The framework also integrates worst-case execution time (WCET) analysis, offering a predictable, efficient, and verifiable deployment solution for deep learning in safety-critical systems.