StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

2026-07-09

AI Summary

This work addresses the throughput bottleneck in large model inference caused by dequantization on GPUs, which incurs high instruction overhead, surges in on-chip traffic, and pipeline stalls. The authors propose the first near-memory dequantization approach by integrating a lightweight dequantization module into the HBM base die. This design leverages sideband tags attached to memory read requests to dynamically select operating modes, enabling “load-and-dequantize” while preserving standard load semantics. Coupled with a customized HBM architecture, on-die dequantization blocks (DQBs), and mixed-precision GEMM acceleration, the method achieves up to a 7.08× speedup in GEMM computation, 90.23% energy reduction, 54.68% lower end-to-end LLM inference latency, and a 2.20× improvement in decoding throughput.

Abstract

As large language models (LLMs) scale, their memory and computation demands have grown substantially, making weight-only quantization a widely adopted technique for reducing model size with minimal accuracy loss. However, on current GPUs, CUDA-core-based dequantization introduces substantial instruction overhead, on-chip traffic, and pipeline stalls, making it a major bottleneck for high-throughput, cloud-scale LLM serving. To address these limitations, we propose StreamDQ, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference. StreamDQ integrates compact DeQuantization Blocks (DQBs) into the base die of high-bandwidth memory (HBM) and performs inline dequantization on standard memory loads. A lightweight sideband tag on each memory read request selects the dequantization mode while preserving conventional load semantics. By relocating dequantization to the memory side, StreamDQ eliminates GPU-side CUDA-core-based dequantization, thereby reducing on-chip traffic on the GPU and avoiding extra HBM write-back and reload of dequantized weights at large batch sizes. Our evaluation shows that StreamDQ achieves up to 7.08$\times$ speedup and 90.23\% lower energy for mixed-precision GEMM, with only 0.127\,mm$^2$ area and 0.355\,W power overhead per DQB in a 12\,nm CMOS process. For end-to-end LLM inference, StreamDQ reduces latency by up to 54.68\% and improves decode throughput by up to 2.20$\times$.

Problem

• weight dequantization
• LLM inference
• memory bottleneck
• GPU overhead
• high-throughput serving

Innovation

• near-memory computing
• weight dequantization
• HBM integration
• LLM inference acceleration
• architectural enhancement