EnerInfer: Energy-Aware On-Device LLM Inference

2026-06-22

AI Summary

This work addresses the high energy consumption and thermal management challenges in on-device large language model inference, which arise from aggressive pursuit of decoding speed. The authors propose a dynamic inference framework that jointly optimizes energy efficiency, throughput, and thermal comfort without requiring per-model profiling or fine-grained power sensors. For the first time, it enables online coordination between energy efficiency and thermal regulation through structure-aware throughput and power prediction, a lightweight finite-horizon thermal model, and a ranking-based online feedback mechanism to dynamically select the optimal NPU and memory frequency configuration under user-perceived quality-of-experience (QoE) constraints. Experiments demonstrate up to 65%, 12%, and 24% improvements in energy efficiency on smartphones, laptops, and development boards, respectively, while consistently meeting QoE requirements.

Abstract

On-device LLM inference is increasingly attractive for privacy-preserving, reliable, and cost-effective deployment, yet its energy and thermal costs remain a critical bottleneck. Existing systems primarily optimize for decoding speed, implicitly assuming that faster execution is always preferable. We show instead that on-device LLM inference often has exploitable configuration slack: modestly lowering NPU and memory frequencies preserves quality of experience (QoE) while substantially improving energy efficiency and reducing heat. Realizing this opportunity in production is challenging. The most energy-efficient NPU/DDR setting varies with the model, inference engine, platform, and runtime conditions, with no stable ranking across configurations. Commercial devices further lack component-level power sensing, and shell temperature evolves with request arrivals, response lengths, and thermal history. To address these challenges, we propose EnerInfer, the first on-device LLM inference framework that jointly manages energy efficiency, throughput, and thermal comfort for LLM workloads. EnerInfer replaces per-model profiling and sensor-heavy control with disaggregated, model-structure-aware prediction and ranking-driven online feedback. It predicts throughput and power for unseen LLMs across NPU/DDR frequency settings, selects QoE-satisfying efficient configurations under runtime interference, and uses lightweight limited-horizon thermal prediction to dynamically switch between energy-optimized and thermally constrained inference. Evaluations on real-world LLMs show that EnerInfer improves energy efficiency by up to 65%, 12%, and 24% on phones, a laptop, and a development board, respectively, without QoE violation.

Problem

• on-device LLM inference
• energy efficiency
• thermal management
• quality of experience
• NPU/DDR frequency

Innovation

• energy-aware inference
• on-device LLM
• thermal management
• configuration optimization
• model-structure-aware prediction