parallel and distributed computing

With AI and high-resolution simulations increasingly driving HPC workloads, parallel I/O performance bottlenecks have grown more complex, while existing optimization tools remain fragmented and difficult to select. Method: We systematically review 131 publications and—employing bibliometric analysis, systematic literature review, and taxonomy modeling—construct the first comprehensive, end-to-end parallel I/O classification framework (a “360° taxonomy”) covering characterization, analysis, and optimization. Our approach integrates cross-platform profiling and tracing tools—including Darshan, Vampir, and Lustre trace—into a unified analytical pipeline. Contribution: We propose the first holistic, cross-layer I/O optimization framework spanning applications, runtime systems, file systems, and hardware; release a structured knowledge graph and open-source classification toolkit; and significantly reduce decision-making overhead in selecting optimization strategies. This work delivers a reusable, scalable methodology for enhancing parallel I/O performance in production HPC environments.

Selecting appropriate parallel programming models for heterogeneous HPC architectures remains challenging due to divergent hardware characteristics and software trade-offs. Method: This paper conducts the first multi-dimensional quantitative comparison of MPI, OpenMP, and CUDA—evaluating architectural adaptability, scalability bottlenecks, development complexity, and domain suitability—and proposes a hybrid programming model selection framework tailored to heterogeneity. The framework integrates communication modeling, memory contention analysis, and GPU kernel optimization for empirical validation. Contribution/Results: Experiments show MPI achieves >92% strong scaling efficiency in distributed, communication-intensive workloads; OpenMP delivers 3.8× speedup on shared-memory loop-parallel tasks; CUDA attains up to 12.5× acceleration on data-parallel kernels; and hybrid strategies yield an average 27% improvement in end-to-end performance. The study provides both theoretical foundations and practical guidelines for optimizing and co-designing programming models in heterogeneous HPC environments.

This work addresses the significant communication bottleneck in multi-GPU training caused by the serial execution of computation and communication. The authors propose a portable runtime mechanism that requires no modifications to vendor libraries or kernels. By dynamically controlling on-chip resource occupancy of compute kernels, elevating the scheduling priority of communication streams, and leveraging shared memory for compute footprint management and cross-GPU resource coordination, the approach effectively enables concurrent execution of computation and collective communication. Evaluated on NVIDIA A40, A100, H100, and AMD MI250X GPUs, the method reduces end-to-end training time by up to 25.5%.

To address CPU-induced communication bottlenecks in heterogeneous supercomputing, this paper systematically proposes and classifies the “GPU-centric communication” paradigm—fully migrating communication control from the CPU to the GPU to enable zero-CPU-intervention, end-to-end GPU-direct communication. Methodologically, it integrates GPUDirect RDMA, CUDA Graphs, programmable NICs (e.g., BlueField DPUs), and GPU kernel-level communication primitives to construct a GPU-native communication control stack. Key contributions include: (1) the first systematic characterization of hardware requirements—GPU memory coherence, NIC programmability, and kernel-driver support—and software challenges—including synchronization, error handling, and programming model adaptation; and (2) a design blueprint for low-latency, high-throughput, ultra-low-CPU-overhead communication infrastructure tailored for HPC and distributed AI training. (149 words)

Existing parallel computing curricula for undergraduate and graduate students often lack a unified, principle-centered pedagogical framework that balances theoretical foundations with practical implementation while ensuring broad applicability. Method: This work develops a systematic lecture note suite grounded in deterministic parallel algorithms, covering core theory (work-time model, efficiency and scalability analysis), mainstream programming models (OpenMP, MPI, pthreads), and C-language implementation—explicitly excluding GPU programming and randomized algorithms to preserve conceptual generality. It integrates visualization-guided explanations, verifiable code examples, and structured programming exercises emphasizing universal performance criteria: execution time, energy consumption, and scalability. Contribution/Results: The resulting self-contained, production-ready lecture notes are accompanied by open-source code and extensible problem sets. They effectively support both formal instruction in parallel and high-performance computing courses and independent learning, enhancing pedagogical coherence and practical accessibility.

This work proposes a novel architecture-aware collective communication algorithm to address the all-to-all communication bottleneck on emerging many-core supercomputers. By holistically considering message size, process count, node topology, and system partitioning, the algorithm optimizes data scheduling and communication pathways. Evaluated on a 32-node system based on Intel Sapphire Rapids processors, the proposed method achieves up to a 3× speedup over state-of-the-art MPI implementations, significantly enhancing communication efficiency for applications such as fast Fourier transforms, matrix transposition, and machine learning workloads.

Data-intensive scientific workflows suffer from inefficient scheduling on NUMA-based HPC systems due to high variance in memory access latency and complex task–data placement constraints. Method: We propose nFlows—the first workflow runtime system supporting NUMA-aware modeling and direct execution on real hardware. It integrates NUMA-aware task scheduling, fine-grained memory binding, heterogeneous memory management (HBM/DRAM), and co-scheduling of accelerators (GPU/FPGA/NIC) with device affinity. It enables seamless validation of scheduling algorithms across simulation and physical platforms while deeply characterizing memory locality and data movement patterns. Results: Experiments demonstrate that nFlows significantly reduces cross-NUMA-domain data access latency, accurately identifies performance bottlenecks, and validates the critical impact of NUMA-aware co-optimization on memory-intensive workflows—achieving substantial end-to-end performance gains.

This study addresses the low utilization of modern GPU computing resources by systematically evaluating the performance, energy efficiency, and resource isolation characteristics of NVIDIA’s Multi-Process Service (MPS) and Multi-Instance GPU (MIG) technologies under concurrent application workloads. The experiments reveal a critical trade-off between MPS’s scheduling flexibility and MIG’s hardware-level isolation: MPS can improve performance by up to 30% and reduce energy consumption by approximately 20% in the absence of memory contention, yet suffers a 30% performance degradation under contention; MIG effectively mitigates resource contention but is constrained by its rigid configuration options and higher overhead. These findings provide empirical foundations for optimizing GPU co-execution strategies driven by application-specific workload characteristics.

General-purpose programs often struggle to achieve efficient parallel execution on GPUs, hindering advances in program synthesis, superoptimization, and array programming. To address this limitation, this work proposes a highly parallel virtual machine architecture tailored for linear algebra computations on GPUs, enabling efficient execution of massively concurrent array programs. By integrating a parallel virtual machine, an array-program execution model, and a scalable evaluation framework, the proposed architecture achieves up to a 147× speedup over serial execution across workloads involving millions of concurrent tasks. This substantial acceleration significantly enhances GPU resource utilization and improves the efficiency of program synthesis pipelines.

This work addresses the complexity and performance instability arising from integrating diverse accelerator APIs—such as CUDA, SYCL, and Triton—and vendor-specific libraries in heterogeneous systems, where differences in abstraction and synchronization mechanisms hinder efficient development. To overcome these challenges, the authors propose a task-based dataflow model that encapsulates accelerator invocations as first-class tasks via Task-Aware APIs (e.g., TACUDA, TASYCL). These tasks are uniformly scheduled by OpenMP/OmpSs-2 runtimes into directed acyclic graphs (DAGs) and coordinated across runtimes through integration with nOS-V for cross-runtime thread collaboration. This approach enables, for the first time, transparent and efficient coexistence of multiple native accelerator programming models within a single application, significantly simplifying multi-API development while enhancing scalability and performance stability across current and future heterogeneous hardware architectures.

This work addresses the challenges traditional high-performance computing (HPC) applications face in adapting to the dynamic, heterogeneous, and resource-volatile nature of cloud environments, particularly in leveraging elastic resources and mitigating performance instability. Building upon the Charm++ asynchronous message-driven runtime system, the authors propose a cloud-native adaptive runtime framework that introduces a novel rate-aware load balancing mechanism to optimize performance across heterogeneous CPU/GPU resources. The framework further extends its resource management module to support low-overhead scheduling of preemptible instances. By effectively alleviating network contention and processor performance fluctuations, the proposed approach significantly enhances the execution efficiency, stability, and cost-effectiveness of HPC applications in cloud settings.