Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backends

2025-11-27

AI Summary

Selecting optimal hardware for deploying small language models (SLMs) in resource-constrained edge computing environments remains challenging due to the lack of systematic, cross-architecture performance and efficiency comparisons. Method: This work conducts the first unified, empirical evaluation—within a consistent experimental framework—of Intel/ARM CPUs, NVIDIA GPUs, and the RaiderChip NPU across mainstream SLMs. We introduce bandwidth-normalized analysis and adopt holistic metrics—including Energy-Delay Product (EDP)—to jointly quantify inference throughput, latency, and energy efficiency. Results: Dedicated NPUs achieve substantially higher throughput and 1–2 orders-of-magnitude lower EDP than general-purpose CPUs. While low-power ARM CPUs exhibit limited raw performance, they demonstrate competitive inference-per-watt efficiency. Our reproducible benchmarking methodology and empirical findings provide actionable, evidence-based guidance for hardware selection in edge-deployed SLM applications.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Edge computing processes data where it is generated, enabling faster decisions, lower bandwidth usage, and improved privacy. However, edge devices typically operate under strict constraints on processing power, memory, and energy consumption, making them unsuitable for large language models (LLMs). Fortunately, Small Language Models (SLMs) offer lightweight alternatives that bring AI inference to resource-constrained environments by significantly reducing computational cost while remaining suitable for specialization and customization. In this scenario, selecting the hardware platform that best balances performance and efficiency for SLM inference is challenging due to strict resource limitations. To address this issue, this study evaluates the inference performance and energy efficiency of commercial CPUs (Intel and ARM), GPUs (NVIDIA), and NPUs (RaiderChip) for running SLMs. GPUs, the usual platform of choice, are compared against commercial NPUs and recent multi-core CPUs. While NPUs leverage custom hardware designs optimized for computation, modern CPUs increasingly incorporate dedicated features targeting language-model workloads. Using a common execution framework and a suite of state-of-the-art SLMs, we analyze both maximum achievable performance and processing and energy efficiency across commercial solutions available for each platform. The results indicate that specialized backends outperform general-purpose CPUs, with NPUs achieving the highest performance by a wide margin. Bandwidth normalization proves essential for fair cross-architecture comparisons. Although low-power ARM processors deliver competitive results when energy usage is considered, metrics that combine performance and power (such as EDP) again highlight NPUs as the dominant architecture. These findings show that designs optimized for both efficiency and performance offer a clear advantage for edge workloads.

Problem

• Evaluates CPU, GPU, NPU performance for Small Language Models on edge devices.
• Compares hardware backends for SLM inference under strict resource constraints.
• Analyzes energy efficiency and speed trade-offs for edge AI deployment.

Innovation

• Compares CPU, GPU, NPU backends for Small Language Models
• Uses common framework to benchmark performance and energy efficiency
• Finds NPUs are optimal for edge due to specialized hardware