FlashAccel: Leveraging High-Bandwidth Flash for High-Throughput LLM Inference

2026-07-11

AI Summary

This work addresses the growing memory bottleneck in large language model (LLM) inference, where the capacity of GPU high-bandwidth memory (HBM) is insufficient to accommodate both model weights and key-value (KV) cache demands. To overcome this limitation, the authors propose the first GPU architecture that synergistically integrates high-bandwidth flash (HBF) storage through a hardware-software co-design approach. The design features an HBF-aware memory management layer, specialized data layouts tailored for weights and KV cache, and a heterogeneous memory coordination mechanism to enable low-latency, high-efficiency data access. Under a strict 100 ms latency constraint, the proposed system achieves a 2.54× improvement in per-GPU inference throughput and a 1.93× gain in energy efficiency compared to conventional HBM-only baselines.

Abstract

Large language model (LLM) inference is increasingly limited by the capacity of High-Bandwidth Memory (HBM) in GPUs, as model weights and KV cache grow rapidly. High-Bandwidth Flash (HBF) provides higher capacity than HBM while retaining comparable bandwidth, making it a promising substrate for capacity-constrained LLM inference. However, its inherently high access latency, low bandwidth utilization, and lack of support for heterogeneous resource management make it difficult to integrate HBF into GPUs for LLM inference. We present FlashAccel, a co-designed system that enables efficient LLM inference using HBF. FlashAccel integrates HBF into HBM-based GPUs, providing architectural support to mitigate access latency. It improves bandwidth utilization through specialized data layouts for both model weights and KV cache, and introduces an HBF-aware storage management layer together with a programming model to organize persistent data in HBF and coordinate heterogeneous memory resources at the system level. Experimental results demonstrate that integrating six HBF stacks into the GPU enables FlashAccel to deliver an average improvement of 2.54$\times$ and 1.93$\times$ in throughput per GPU and energy efficiency over the HBM-only GPU under 100ms latency constraint, respectively.

Problem

• LLM inference
• High-Bandwidth Memory
• High-Bandwidth Flash
• memory capacity
• heterogeneous resource management

Innovation

• High-Bandwidth Flash
• LLM inference
• heterogeneous memory management
• KV cache optimization
• bandwidth utilization