Exploring High-Bandwidth Flash for Modern LLM Inference: Opportunities and Challenges

2026-08-13

AI Summary

This study addresses the GPU memory bottleneck in Large Language Model (LLM) inference by systematically evaluating the feasibility of High-Bandwidth Flash (HBF) as main memory extension. Through comprehensive system modeling and multi-scenario performance analysis under mixed read-write workloads, we investigate the associated benefits and limitations. Results demonstrate that HBF significantly enhances system throughput and service flexibility while effectively reducing GPU hardware requirements. However, practical deployment necessitates prioritizing read bandwidth provisioning and addressing endurance challenges. Ultimately, this work establishes a novel paradigm for overcoming the storage wall in LLM inference and provides critical engineering guidelines for integrating flash-based memory architectures into next-generation AI serving systems.

Abstract

This work investigates the potential benefits and technical challenges of using high-bandwidth flash (HBF) for large language model (LLM) inference. HBF has gained increasing attention as a promising solution to mitigate memory-capacity bottlenecks in modern LLM-serving systems, but its benefits and challenges remain largely uninvestigated. To address this gap, we thoroughly analyze HBF-based LLM-serving systems under diverse system configurations and operating scenarios in which HBF serves as a main GPU-memory component to handle both reads and writes. Our analysis shows that, despite its limited write performance, HBF can significantly improve the batch size, throughput, and flexibility of LLM-serving systems while reducing the minimum GPU requirements, but realizing these benefits critically depends on sustaining HBM-comparable read bandwidth and requires significant endurance improvements.

Problem

• High-Bandwidth Flash
• LLM Inference
• Memory Bottleneck
• Storage System

Innovation

• High-Bandwidth Flash
• LLM Inference
• Memory Capacity Bottleneck
• Read Bandwidth
• Endurance