NITRO: High-Performance 3D NAND Flash-Based In-Storage Computing with Enhanced Activation Dataflow

2026-08-12

AI Summary

This work addresses the performance bottleneck in large language model (LLM) inference caused by storing activation values in high-latency TLC NAND flash within conventional NAND-based processing-in-memory (PIM) architectures. To overcome this limitation, the authors propose a high-performance architecture that introduces a high-speed DRAM subsystem to cache activation values and incorporates a distributed dataflow mechanism tailored for NAND-PIM arrays along with an intra-plane parallel data mapping strategy. By migrating activation buffering from slow NAND to DRAM, the design effectively alleviates write latency and bandwidth constraints, significantly enhancing computational parallelism and overall efficiency. Experimental results demonstrate that the proposed approach reduces inference latency by up to 85% compared to the baseline, substantially improving LLM execution efficiency on high-density storage hardware.

Abstract

In-storage computing (ISC) is considered a next-generation memory architecture for its ability to relieve the data bottleneck between the host and the memory. While the required resources of large language models (LLMs) have increased significantly in recent years, the memory density has not scaled accordingly. Recently, several works have studied NAND flash-based processing-in-memory (NAND-PIM) schemes to exploit the high density of the memory. However, they do not address the dataflow/buffer for the intermediate values, so a simple method is to deal with the values in the slow flash memory array. To overcome such a limitation, we propose a high-performance NAND flash-based ISC architecture with enhanced activation buffering. Instead of using the very slow flash memory array for the intermediate values, our architecture buffers the values in a fast DRAM subsystem. This approach effectively handles the high-latency penalties when activations are programmed into slower TLC NAND flash. We also introduce a distributed dataflow approach for the NAND-PIM array. This approach maximizes computational parallelism by employing efficient intra-plane data mapping. The results show that our proposed architecture achieves significant performance improvements, reducing the inference latency by up to 85% compared to the baseline.

Problem

• in-storage computing
• NAND flash
• activation dataflow
• intermediate values
• memory bottleneck

Innovation

• In-storage computing
• NAND flash
• Activation buffering
• Dataflow optimization
• Processing-in-memory