EmuGEMM: Fused Tensor Core Kernels for Precision Emulation in Matrix Multiplication

2026-06-24

AI Summary

This work addresses the performance limitations of high-precision matrix multiplication on modern GPUs, which stem from the inherent tension between the low-precision efficiency of Tensor Cores and memory bandwidth constraints. Specifically, the Ozaki scheme suffers from frequent writes of intermediate results to global memory, degrading performance. The paper presents the first hardware-level fusion of multi-stage Ozaki Scheme I/II computations, leveraging integer Tensor Cores on NVIDIA Hopper and Blackwell architectures to implement a fully fused GEMM kernel that entirely eliminates redundant global memory traffic. Maintaining comparable numerical accuracy, the proposed approach achieves 1.4–1.7× speedup over cuBLAS TF32 for real-valued GEMM and 2.3–5.5× for complex ZGEMM. It attains 1,639 TOP/s (83% of INT8 peak) on Hopper and 3,654 TOP/s (81% of INT8 peak) on Blackwell.

Abstract

Modern GPUs devote an increasing silicon budget to low-precision matrix-multiplication units, widening the precision-throughput gap for scientific computing workloads. Ozaki Schemes I and II offer an alternative by reconstructing high-precision general matrix multiplication (GEMM) from low-precision operations, yet existing implementations leave substantial performance untapped. In particular, intermediate results are repeatedly materialized in global memory, making data movement the dominant bottleneck. We present EmuGEMM, fused integer Tensor Core kernels for NVIDIA Hopper and Blackwell GPUs that eliminate redundant memory round-trips in both Ozaki schemes. Using Scheme I, EmuGEMM sustains up to 1,639 Top/s on Hopper (83% of INT8 peak) and 3,654 Top/s on Blackwell (81%). For large matrices, EmuGEMM surpasses cuBLAS TF32 throughput by up to 1.4x on Hopper and 1.7x on Blackwell, at comparable accuracy. Using Scheme II, EmuGEMM extends to complex arithmetic and outperforms cuBLAS ZGEMM by up to 2.3x on Hopper and 5.5x on Blackwell.

Problem

• precision-throughput gap
• general matrix multiplication
• data movement bottleneck
• low-precision operations
• scientific computing

Innovation

• EmuGEMM
• Tensor Core
• precision emulation
• fused kernels
• Ozaki scheme