Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

2026-07-06

AI Summary

This study addresses significant energy-efficiency bottlenecks in large language model (LLM) inference on mobile devices, particularly when leveraging NPUs. It presents the first fine-grained, backend-aware energy attribution analysis for mobile LLM inference, uncovering substantial efficiency losses stemming from mismatches between framework-level strategies and hardware characteristics—specifically, NPUs exhibit high efficiency during the prefill phase but severe inefficiency during decoding. To investigate this, the authors develop PowerBench, a profiling tool that integrates multiple frameworks (e.g., llama.cpp, GENIE) and hardware backends. Through empirical evaluation with customized operators, quantization schemes, and thread scheduling optimizations, they identify performance gaps of up to 10× and energy waste of up to 40%. Their proposed energy-efficient configurations reduce NPU energy consumption by as much as 54.8%.

Abstract

Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks (e.g., llama.cpp, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond traditional device-level measurements. Our study yields three critical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators due to divergent offloading and quantization strategies. (2) We identify a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding. This is driven by the NPU's preference for large, fixed-shape workloads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substantial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste. Leveraging these findings, we present an energy-oriented best-practice configuration for mobile LLM inference. We estimate that this configuration could reduce energy consumption by up to 54.8% on the NPU backend across three datasets.

Problem

• LLM inference
• mobile devices
• NPU efficiency
• hardware bottlenecks
• energy consumption

Innovation

• mobile LLM inference
• NPU efficiency
• cross-layer profiling
• PowerBench
• energy optimization