KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing

2025-12-03

AI Summary

To address DRAM capacity and bandwidth bottlenecks in long-context LLM inference on edge devices—where KV cache size often exceeds model weight storage—this work proposes the first DRAM-free in-flash computing (IFC) architecture. Methodologically, it (1) unifies model weights and KV cache storage within compute-enabled 3D NAND flash; (2) introduces head-group parallelism and page-level KV cache mapping to enhance access density and efficiency; and (3) develops a spatial exploration framework that automatically optimizes hybrid weight–KV storage layouts. Evaluation demonstrates geometric mean speedups of 1.98×, 1.94×, and 2.05× over DRAM-dependent IFC baselines for 128-, 1K-, and 10K-token contexts, respectively. Crucially, it achieves, for the first time on edge devices, stable 100K-token context inference without memory overflow—eliminating DRAM reliance entirely.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Deploying large language models (LLMs) on edge devices enables personalized agents with strong privacy and low cost. However, with tens to hundreds of billions of parameters, single-batch autoregressive inference suffers from extremely low arithmetic intensity, creating severe weight-loading and bandwidth pressures on resource-constrained platforms. Recent in-flash computing (IFC) solutions alleviate this bottleneck by co-locating weight-related linear computations in the decode phase with flash, yet still rely on DRAM for the key-value (KV) cache. As context length grows, the KV cache can exceed model weights in size, imposing prohibitive DRAM cost and capacity requirements. Attempts to offload KV cache to flash suffer from severe performance penalties. We propose KVNAND, the first DRAM-free, IFC-based architecture that stores both model weights and KV cache entirely in compute-enabled 3D NAND flash. KVNAND addresses the fundamental performance challenges of flash under intensive KV cache access by leveraging IFC for all memory-bound operations to reduce data transfer overhead, introducing head-group parallelism to boost throughput, and employing page-level KV cache mapping to align token access patterns with flash organization. In addition, we propose a design space exploration framework that evaluates discrete and compact KVNAND variants to balance weight and KV placement, automatically identifying the optimal design trade-off. These techniques mitigate latency, energy, and reliability concerns, turning flash into a practical medium for long-context KV storage. Evaluations on MHA 7B and GQA 70B LLMs show that KVNAND achieves 1.98( imes)/1.94( imes)/2.05( imes) geomean speedup at 128/1K/10K-token contexts compared to DRAM-equipped IFC designs and addresses out-of-memory failures at 100K context length.

Problem

• Eliminates DRAM dependency for KV cache in on-device LLMs
• Reduces data transfer overhead via in-flash computing for memory operations
• Optimizes flash access patterns for long-context KV storage efficiency

Innovation

• Stores weights and KV cache entirely in compute-enabled 3D NAND flash
• Uses head-group parallelism and page-level mapping to boost throughput
• Employs design space exploration to optimize weight and KV placement