HetGPU: The pursuit of making binary compatibility towards GPUs

2025-06-19

AI Summary

Heterogeneous GPUs exhibit severe binary incompatibility due to divergent instruction sets, execution models, and driver stacks. To address this, we present the first cross-vendor binary-compatible system supporting NVIDIA, AMD, Intel, and Tenstorrent GPUs. Our approach comprises three key components: (1) an architecture-agnostic intermediate representation (IR) that unifies SIMT and MIMD execution semantics; (2) a dynamic binary translation runtime coupled with state-aware serialization for real-time migration; and (3) a unified abstraction layer ensuring consistent cross-platform semantics for threads, memory, and synchronization. Crucially, our system enables direct binary migration across all four GPU architectures without source or binary modification. Evaluation shows migration overhead under 8% and average performance degradation below 12%, effectively breaking down long-standing hardware-software co-design barriers across GPU vendors.

Technology Category

• Machine Learning: Hardware-aware ML
• Intelligent Robots: Multi-Robot Systems
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Heterogeneous GPU infrastructures present a binary compatibility challenge: code compiled for one vendor's GPU will not run on another due to divergent instruction sets, execution models, and driver stacks . We propose hetGPU, a new system comprising a compiler, runtime, and abstraction layer that together enable a single GPU binary to execute on NVIDIA, AMD, Intel, and Tenstorrent hardware. The hetGPU compiler emits an architecture-agnostic GPU intermediate representation (IR) and inserts metadata for managing execution state. The hetGPU runtime then dynamically translates this IR to the target GPU's native code and provides a uniform abstraction of threads, memory, and synchronization. Our design tackles key challenges: differing SIMT vs. MIMD execution (warps on NVIDIA/AMD vs. many-core RISC-V on Tenstorrent), varied instruction sets, scheduling and memory model discrepancies, and the need for state serialization for live migration. We detail the hetGPU architecture, including the IR transformation pipeline, a state capture/reload mechanism for live GPU migration, and an abstraction layer that bridges warp-centric and core-centric designs. Preliminary evaluation demonstrates that unmodified GPU binaries compiled with hetGPU can be migrated across disparate GPUs with minimal overhead, opening the door to vendor-agnostic GPU computing.

Problem

• Enabling binary compatibility across diverse GPU vendors
• Dynamic translation of GPU intermediate representation to native code
• Bridging execution model differences for seamless GPU migration

Innovation

• Compiler emits architecture-agnostic GPU IR
• Runtime dynamically translates IR to native code
• Abstraction layer bridges divergent GPU architectures