Integrating High Performance In-Memory Data Streaming and In-Situ Visualization in Hybrid MPI+OpenMP PIC MC Simulations Towards Exascale

2025-12-03

AI Summary

Traditional file-based I/O severely limits scalability and efficiency in exascale-capable, billion-particle particle-in-cell (PIC) Monte Carlo simulations for fusion energy research. Method: This paper introduces a novel simulation–analysis paradigm integrating in-memory data streaming with in situ visualization. It leverages the ADIOS2 SST engine to establish a low-overhead memory-streaming architecture enabling interruption-free checkpointing and real-time visualization; further enhanced by MPI+OpenMP hybrid parallelism, openPMD streaming APIs, and multi-dimensional performance profiling tools (gprof, perf, IPM, Darshan). Contribution/Results: Experiments demonstrate substantial reductions in total simulation time, significant improvements in I/O throughput and resource utilization, and robust scalability. The approach establishes an extensible, low-latency framework for exascale co-simulation, advancing turbulent plasma dynamics and confinement studies in magnetic fusion research.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Efficient simulation of complex plasma dynamics is crucial for advancing fusion energy research. Particle-in-Cell (PIC) Monte Carlo (MC) simulations provide insights into plasma behavior, including turbulence and confinement, which are essential for optimizing fusion reactor performance. Transitioning to exascale simulations introduces significant challenges, with traditional file input/output (I/O) inefficiencies remaining a key bottleneck. This work advances BIT1, an electrostatic PIC MC code, by improving the particle mover with OpenMP task-based parallelism, integrating the openPMD streaming API, and enabling in-memory data streaming with ADIOS2's Sustainable Staging Transport (SST) engine to enhance I/O performance, computational efficiency, and system storage utilization. We employ profiling tools such as gprof, perf, IPM and Darshan, which provide insights into computation, communication, and I/O operations. We implement time-dependent data checkpointing with the openPMD API enabling seamless data movement and in-situ visualization for real-time analysis without interrupting the simulation. We demonstrate improvements in simulation runtime, data accessibility and real-time insights by comparing traditional file I/O with the ADIOS2 BP4 and SST backends. The proposed hybrid BIT1 openPMD SST enhancement introduces a new paradigm for real-time scientific discovery in plasma simulations, enabling faster insights and more efficient use of exascale computing resources.

Problem

• Addresses I/O bottlenecks in exascale plasma simulations.
• Enables real-time data analysis and in-situ visualization.
• Improves computational efficiency and storage utilization.

Innovation

• OpenMP task-based parallelism for particle mover
• ADIOS2 SST engine for in-memory data streaming
• openPMD API for real-time in-situ visualization