MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocation

2026-05-23

AI Summary

This work addresses the challenge in deep learning quantization of simultaneously preserving accuracy, dynamic range, and energy efficiency across both training and inference. The authors propose MX-SAFE, an adaptive microscaling format that unifies support for training and direct conversion to inference by dynamically switching between FP8 E2M5 and FP5 E3M2 representations at runtime, thereby optimizing the allocation of exponent and mantissa bitwidths. Coupled with a tiled hardware architecture and a dedicated accelerator, MX-SAFE substantially reduces the overhead of re-quantization. Experiments demonstrate that MX-SAFE improves average inference accuracy by 0.05% and 11.1% over MXFP8 E2M5 and E4M3, respectively, and enhances full-training accuracy by 3.55% and 3.57%. Moreover, it achieves a 24.9% reduction in total energy consumption while maintaining BF16-level accuracy.

Abstract

As the demand for deep learning grows, cost reduction through quantization has become essential for both training and inference. In 2022, the Open Compute Project (OCP) consortium standardized narrow precision formats for deep learning, called the microscaling (MX) format. The MX format is a hardware-friendly dynamic quantization scheme that effectively reduces the data size by sharing an 8-bit exponent across multiple operands. The MX format can be categorized into two types with their own strengths: (i) MXINT which focuses on a high precision consisting only of mantissa bits and (ii) MXFP which focuses on a wider dynamic range by allowing local exponent bits. In this work, we present a versatile MXFP format, called MX-SAFE (MXSF in short), that adaptively uses two modes, i.e., a wider mantissa mode (FP8 E2M5) and a subnormal FP mode (FP5 E3M2), to support both training and direct-cast inference. Furthermore, we propose a tile-based block design to increase hardware efficiency by reducing the burden of re-quantization process during the training with the MXSF format. Owing to the use of the proposed MXSF format, 0.05%/11.1% and 3.55%/3.57% improvements in accuracy, on average, for inference/full-training compared to MXFP8 E2M5 and MXFP8 E4M3 are observed, respectively. Moreover, we present a training-inference accelerator that supports the MXSF format and it achieves similar accuracy to the BF16 baseline while using 24.9% less total energy consumption.

Problem

• quantization
• microscaling
• deep learning
• training-inference efficiency
• precision-dynamic range trade-off

Innovation

• microscaling
• adaptive quantization
• FP8
• hardware-efficient training
• energy-efficient inference