DiOMP-Offloading: Toward Portable Distributed Heterogeneous OpenMP

2025-06-03

AI Summary

To address portability bottlenecks in HPC arising from GPU heterogeneity and challenging distributed memory management, this paper introduces DiOMP, a distributed OpenMP framework. Methodologically, DiOMP features a novel unified runtime that integrates OpenMP target offloading with Partitioned Global Address Space (PGAS) semantics, enabling both symmetric and asymmetric GPU memory allocation. It further incorporates OMPCCL—a lightweight, portable collective communication layer—and is implemented via LLVM/OpenMP extensions, supporting GASNet-EX and GPI-2 communication backends across NVIDIA, AMD, and Grace Hopper platforms. Evaluation on A100, Grace Hopper, and MI250X systems demonstrates significant performance improvements for applications including matrix multiplication and MiniMod, while achieving strong scalability and programming simplicity.

Technology Category

• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

As core counts and heterogeneity rise in HPC, traditional hybrid programming models face challenges in managing distributed GPU memory and ensuring portability. This paper presents DiOMP, a distributed OpenMP framework that unifies OpenMP target offloading with the Partitioned Global Address Space (PGAS) model. Built atop LLVM/OpenMP and using GASNet-EX or GPI-2 for communication, DiOMP transparently handles global memory, supporting both symmetric and asymmetric GPU allocations. It leverages OMPCCL, a portable collective communication layer compatible with vendor libraries. DiOMP simplifies programming by abstracting device memory and communication, achieving superior scalability and programmability over traditional approaches. Evaluations on NVIDIA A100, Grace Hopper, and AMD MI250X show improved performance in micro-benchmarks and applications like matrix multiplication and Minimod, highlighting DiOMP's potential for scalable, portable, and efficient heterogeneous computing.

Problem

• Manages distributed GPU memory in HPC systems
• Ensures portability across heterogeneous architectures
• Simplifies programming by abstracting device memory and communication

Innovation

• Unifies OpenMP offloading with PGAS model
• Uses GASNet-EX or GPI-2 for communication
• Leverages OMPCCL for portable collective communication