A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving

2026-08-12

AI Summary

This work demonstrates that high-bandwidth flash (HBF) cannot serve as a plug-and-play replacement for SSDs in large language model KV caching. Despite its superior bandwidth, HBF suffers from thermal throttling and TLC wear under write-intensive KV access patterns, leading to 2–5.5× higher end-to-end latency and 1.1–2.7× lower SLO-compliant throughput. Through full-stack simulation (extending TokenSim), real-world production workloads (Qwen-Bailian), realistic hardware configurations (H100/B200), and a 3D-ICE thermal model, this study presents the first systematic evaluation of HBF in KV-centric inference. The findings suggest that HBF should be deployed not as a general-purpose cache backend, but as a selective, reuse-aware, write-budget-constrained, and thermally coordinated tier within the storage hierarchy.

Abstract

High-Bandwidth Flash (HBF) stacks NAND behind a wide, package-local interface, giving flash-scale capacity with far better read latency and bandwidth than an SSD. This makes it tempting to keep an SSD-style Mooncake KV-offloading stack and swap only the backing tier for HBF. We test that substitution with an extended TokenSim, four complete two-hour Qwen-Bailian production traces, five dense and mixture-of-experts models, and H100/B200 profiles. Serving gets worse, not better, and a cost-benefit model explains why. A faster far tier helps only when read I/O is the serving bottleneck, reads outweigh writes, and delivered bandwidth is sustainable. All three must hold together, and transient KV fails every one. The package trade that buys flash costs GPU near-tier capacity and bandwidth, so average end-to-end latency rises 2--5.5x and maximum SLO goodput falls 1.1--2.7x across H100 and B200. Serving is almost insensitive to HBF's own read/write latency, and base-die near-memory compute does not raise the flash tier's share of the critical path. The two-tier hierarchy keeps reuse in the near tier and hands HBF a write-heavy stream, so writes outnumber reads on every trace. A 3D-ICE model shows that stream drives the stack to its thermal limit well below peak bandwidth, and a TLC tier wears out sooner than a capacity-matched SSD pool. The faster device yields a slower system because the package gives up more than the medium returns. HBF is not the problem; using it as a faster SSD for transient KV is. It belongs in serving as a selective, reuse-aware, write-budgeted, and thermally coordinated tier, not as a drop-in SSD replacement.

Problem

• High-Bandwidth Flash
• KV-Centric LLM Serving
• Storage Hierarchy
• Transient KV Cache
• Write-Heavy Workload

Innovation

• High-Bandwidth Flash
• KV-Centric LLM Serving
• Storage Hierarchy
• Write-Heavy Workload
• Thermal-Aware Design