Potential Applications of HBF in LLM Serving Systems

2026-08-13

AI Summary

This work addresses the memory capacity bottleneck in large language model (LLM) serving caused by the limited high-bandwidth memory (HBM), which struggles to accommodate growing model weights, KV caches, and multiple model variants. For the first time, it systematically explores high-bandwidth flash (HBF) as a capacity-extension layer for HBM, preserving high-bandwidth compute pathways while integrating HBF into the GPU memory hierarchy. The authors design model state residency policies tailored to the read-heavy, write-light access patterns of LLM inference, supported by system-level modeling. Experimental results demonstrate that this approach significantly increases the number of expert replicas in mixture-of-experts (MoE) architectures, reduces loading overhead in multi-model serving, and enables replication of popular models, thereby substantially improving overall serving efficiency.

Abstract

LLM serving is increasingly constrained by memory capacity as model weights, KV caches, and the number of served model variants continue to grow. This report examines High-Bandwidth Flash (HBF) as a capacity-oriented extension to HBM-based serving systems. We first discuss how HBF can be integrated into the GPU memory hierarchy without undermining the bandwidth expected by the compute die. We then model the system-level value of added capacity as expanded residency for read-mostly model-state objects. Under this view, HBF can improve MoE serving by enabling more expert replicas and can improve multi-model serving by reducing model loading and supporting hot-model replication. Our simulation results show that these benefits depend on preserving the HBM-resident execution path while using HBF to expand the resident set of model weights.

Problem

• LLM serving
• memory capacity
• KV cache
• model variants
• HBM

Innovation

• High-Bandwidth Flash
• LLM serving
• memory hierarchy
• model residency
• MoE