Improved Scaling for Fast Mode of Ozaki Scheme II

2026-06-27

AI Summary

This work addresses a critical limitation in Ozaki Scheme II’s fast mode, where matrix scaling lacks scale invariance, causing variations in integer bit-width that degrade numerical accuracy or lead to Chinese Remainder Theorem (CRT) reconstruction failure. To resolve this, the authors propose a scale-invariant scaling formula derived from the uniqueness condition of CRT and the Cauchy–Schwarz inequality. This approach guarantees CRT uniqueness without introducing additional computational overhead. Evaluated on INT8 low-precision integer matrix multiplication, the method achieves high-fidelity results, demonstrating accuracy comparable to the accurate mode and throughput approaching that of the fast mode on NVIDIA GH200 GPUs—significantly outperforming the original scheme’s trade-off between precision and performance.

Abstract

Ozaki scheme II emulates high-precision matrix multiplication using low-precision integer matrix operations based on the Chinese remainder theorem (CRT). It first scales the high-precision matrices to convert them into integer matrices. For this scaling step, Ozaki scheme II provides two modes: accurate mode, which uses INT8 matrix multiplication to estimate scaling factors, and fast mode, which applies the Cauchy--Schwarz inequality at lower computational cost. We show that the existing formula lacks scale invariance; multiplying the input matrices by a constant changes the effective bit width of the integer matrices in the scaling step, causing accuracy degradation or CRT recovery failure. To address this, we propose a revised scaling formula derived from the CRT uniqueness condition via the Cauchy--Schwarz inequality. The proposed formula is scale-invariant by construction, guarantees that the CRT uniqueness condition is always satisfied, and introduces no additional overhead over the original fast mode. Experiments on an NVIDIA GH200 GPU show that the proposed method achieves accuracy comparable to that of accurate mode while maintaining throughput comparable to that of fast mode. In the accuracy--throughput trade-off, the proposed method overcomes the accuracy limitation of fast mode and the throughput constraint of accurate mode, offering a superior accuracy and performance.

Problem

• scale invariance
• Ozaki scheme
• Chinese remainder theorem
• matrix multiplication
• low-precision computing

Innovation

• scale invariance
• Chinese remainder theorem
• Ozaki scheme
• low-precision integer matrix multiplication
• Cauchy-Schwarz inequality