Problem
Research questions and friction points this paper is trying to address.
on-device LLM inference
activation sparsity
compute-I/O overlap
memory usage
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
🤖 AI Summary
为解决移动和嵌入式设备上大模型推理时内存不足的问题，LeanStream通过一种推测与精炼的流处理框架，优化了计算与存储I/O之间的重叠，减少了内存使用并提高了吞吐量。
📝 Abstract
On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload weights to SSD or flash storage, but face a fundamental systems trade-off: accurate sparse execution decisions require the latest context, whereas efficient computation-I/O overlap requires early prediction. As a result, existing designs either serialize execution or incur redundant weight fetches, extra computation, and large cache overheads. We present LeanStream, a streaming speculate-and-refine framework for efficient on-device LLM inference. LeanStream progressively refines computation, loading, and cache-retention priorities using partial GPU results, enabling fine-grained overlap between GPU execution and storage I/O. We implement LeanStream on both mobile and embedded platforms. Compared with prior on-device LLM inference systems, LeanStream reduces memory usage by 4.8$\times$ to 7.5$\times$ at the best throughput achieved by prior work, while further improving token generation throughput by 1.6$\times$ to 2.1$\times$.
Innovation
Methods, ideas, or system contributions that make the work stand out.
speculate-and-refine
streaming framework
on-device LLM inference
fine-grained overlap
memory usage reduction
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
