ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

2026-07-16

AI Summary

Traditional CPUs struggle to efficiently support extremely low-bit (1/2/4-bit) general matrix multiplication (GEMM) due to poor hardware adaptability, high costs, and fragmented precision combinations. This work proposes ExaGEMM—the first hardware-software co-design framework for native low-bit GEMM on CPUs—introducing an innovative in-register lookup table execution mechanism coupled with workload-aware exploration. ExaGEMM requires only lightweight selection and feed-forward logic added to existing SIMD pipelines. By integrating parametric kernel generation, SIMD ISA extension modeling, an analytical cost model, and gem5 simulation, ExaGEMM achieves an average 13.29× latency reduction over pure software baselines across representative machine learning models, demonstrating the critical value of workload-driven design in mixed-precision large models.

Abstract

Low-bit GEMM is increasingly central to efficient ML inference, yet very-low-bit execution remains a poor fit for conventional CPUs. Practical deployment spans fragmented regimes-from 1/2/4-bit weights to varying activation precision-whose feasibility, reuse opportunity, and support cost differ under fixed SIMD and register-file budgets, making lightweight CPU support selection a first-class design problem. We present ExaGEMM, a workload-aware codesign and exploration framework for CPU-native low-bit GEMM via register-resident LUT execution. The key insight is that existing SIMD datapaths already cover table generation and accumulation; the only new hardware is an in-register select/feed mechanism with explicitly modeled cost. ExaGEMM co-explores parameterized kernels and lightweight SIMD ISA support using analytical models of register feasibility, compute cost, memory traffic, and hardware overhead, pruning the candidate space by 99.2% before simulation. It then identifies non-dominated support points and generates ISA specs, gem5 patches, and GEMM kernels for validation. Across representative ML models and CPU targets, ExaGEMM improves latency by 13.29x over software-only baselines, while showing that workload-aware frontier selection is especially important for mixed-precision LLM workloads.

Problem

• low-bit GEMM
• CPU inference
• SIMD
• register file
• mixed-precision

Innovation

• low-bit GEMM
• in-register computing
• SIMD ISA extension
• associative LUT execution
• codesign exploration