High-Performance Resilient Multi-GPU Hybrid Particle-in-Cell Monte Carlo Simulations at Scale

2026-06-26

AI Summary

This work addresses the challenges of scalability, load balancing, and fault tolerance in large-scale particle-in-cell Monte Carlo (PIC-MC) simulations for plasma physics on heterogeneous multi-GPU systems. We present the first unified, high-performance, fault-tolerant PIC-MC framework supporting both NVIDIA and AMD GPUs, integrating an MPI+OpenMP hybrid parallel model, a dynamic load-balancing mechanism, and standardized I/O based on openPMD/ADIOS2. The framework innovatively combines BP4 file-based checkpointing with SST in-memory streaming to enable elastic execution and seamless recovery. Strong and weak scaling up to 800 GPUs is demonstrated on Frontier, MN5, and LUMI-G systems, enabling efficient and robust simulations with tens of billions of particles while supporting in situ analysis and visualization.

Abstract

The increasing demand for high-performance computing in plasma physics has driven scalable and resilient simulation methods capable of efficiently exploiting modern multi-GPU architectures. This work extends a portable hybrid MPI+OpenMP implementation of BIT1, focusing on high-performance resilience for accelerated Particle-in-Cell (PIC) Monte Carlo (MC) simulations under both uniform and non-uniform load conditions. Scalable particle load balancing and robust checkpoint/restart mechanisms across Nvidia and AMD accelerators are integrated with standardized I/O using openPMD and ADIOS2. This leverages BP4 for high-performance file-based checkpointing and SST for in-memory data streaming, enabling efficient data movement, resilient large-scale execution, seamless continuation from existing checkpoints, and effective handling of computational and I/O workloads. Advanced HPC profiling and tracing tools, including Nvidia Nsight Systems and AMD ROC-Profiler with Perfetto, provide detailed insights into computation, communication, and system-level behavior for optimization. Performance results on Frontier (OLCF-5), MN5, and LUMI-G demonstrate strong and weak scaling up to 800 GPUs, validating the framework for large-scale PIC MC simulations, while in-situ analysis and visualization using scalable I/O further enhance scientific insight without interrupting multi-GPU execution on current and future exascale systems.

Problem

• Particle-in-Cell
• Monte Carlo
• multi-GPU
• resilience
• exascale

Innovation

• multi-GPU
• resilient checkpointing
• load balancing
• in-situ analysis
• exascale simulation