HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators

2026-06-29

AI Summary

This work addresses the inefficiency in homogeneous hardware systems for large language model inference, where the compute-intensive prefill phase underutilizes high-bandwidth memory (HBM) bandwidth, while the memory-intensive decode phase suffers from cost-ineffective resource allocation. To overcome this, the authors propose a cross-vendor heterogeneous memory architecture that offloads prefill and decode computations to GDDR- and HBM-equipped accelerators, respectively. The design incorporates three key techniques: phase-aware quantization, a compute-transfer pipeline that overlaps KV cache transmission with prefill computation, and lazy dequantization that defers decompression until the decode stage. Evaluated on Qwen3-series models under realistic workloads, the system achieves up to 3.2× higher throughput and 4.8× better throughput per dollar, with negligible degradation in generation quality.

Abstract

LLM inference comprises a compute-bound prefill phase and a memory-bound decode phase, and recent systems disaggregate them onto separate hardware. Yet today's datacenter GPUs rely on costly HBM whose bandwidth sits almost entirely idle during prefill. LLM serving across memory-heterogeneous accelerators (MemHA) pairs GDDR-based accelerators for prefill with HBM-based GPUs for decode, promising lower cost without sacrificing performance. Pushed to its most economical form, MemHA serving is inherently cross-vendor, since the best-suited chip for each phase may come from a different vendor. This breaks two assumptions that single-vendor disaggregation takes for granted -- a KV format both ends consume natively, and a shared software stack. We present \textbf{HMA-Serve}, a MemHA-centric disaggregated serving system pairing GDDR-based accelerators for prefill with HBM-based GPUs for decode efficiently. HMA-Serve achieves this through (1) phase-wise quantization, applying vendor-native low precision for high-throughput prefill while keeping decode in high-precision BF16, (2) a compute-transfer pipeline that overlaps each layer's KV cache transfer with later-layer prefill to reduce time-to-first-token (TTFT), and (3) deferred dequantization, shipping raw quantized bytes and reconstructing them lazily on the decode GPU to reduce network bandwidth and HBM usage. Across four Qwen3 models (4B--32B) and three production traces, HMA-Serve delivers up to $3.2\times$ higher goodput than state-of-the-art memory-homogeneous methods and $4.8\times$ higher goodput-per-dollar, with no measurable loss on generation-quality benchmarks.

Problem

• LLM serving
• memory-heterogeneous accelerators
• disaggregation
• cross-vendor
• KV cache

Innovation

• disaggregated LLM serving
• memory-heterogeneous accelerators
• phase-wise quantization
• compute-transfer pipeline
• deferred dequantization