NPUEval: Optimizing NPU Kernels with LLMs and Open Source Compilers

2025-07-18

AI Summary

The weak NPU programming ecosystem, LLMs’ lack of domain-specific code priors, and the heavy reliance of kernel optimization on expert knowledge hinder efficient NPU software development. Method: This work introduces the first benchmark for NPU kernel generation—comprising 102 common operators—integrating LLM-based code generation (e.g., DeepSeek-R1), open-source compiler analysis (LLVM/MLIR) for vectorization, and hardware-level functional and performance validation on AMD NPUs. It incorporates domain-specific C++ vector intrinsics and compiler feedback to enable end-to-end generation–compilation–execution evaluation. Contribution/Results: Experiments reveal that state-of-the-art LLMs achieve >50% vectorization rates on select operators but only ~10% on average, exposing critical limitations in generating accelerator-optimized code. This benchmark establishes a reproducible, extensible evaluation infrastructure for AI-for-Code research in hardware–software co-optimization.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Hardware-aware ML
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Neural processing units (NPUs) are gaining prominence in power-sensitive devices like client devices, with AI PCs being defined by their inclusion of these specialized processors. Running AI workloads efficiently on these devices requires libraries of optimized kernels. Creating efficient kernels demands expertise in domain-specific C++ with vector intrinsics and in-depth knowledge of the target architecture. Unlike GPU programming, which has had years to mature, NPU programming is new, with smaller and more fragmented developer communities across hardware platforms. This fragmentation poses a challenge when utilizing LLMs to assist in writing NPU kernels, as domain-specific optimized code examples are underrepresented in LLM pre-training data. In this paper we introduce NPUEval -- a benchmark for writing and evaluating NPU kernels, consisting of 102 common operators for machine learning workloads. We evaluate LLM generated code on actual hardware based on both functional correctness and vectorization efficiency using open source compiler tools targeting the AMD NPU. We evaluate a range of state-of-the-art LLMs with a mix of proprietary and open-weight models. Latest reasoning models like DeepSeek R1, show promising results achieving out-of-the-box 50%+ vectorization on select kernels. However, the average score across the entire dataset remains roughly 10% even with compiler feedback and vectorized kernel examples -- showing that this is a challenging dataset even for frontier models. The dataset and evaluation code will be released with a permissive open source license, providing an essential benchmark for advancing research in code generation and NPU kernel optimization.

Problem

• Optimizing NPU kernels for AI workloads efficiently
• Addressing fragmented NPU developer communities and expertise gaps
• Evaluating LLM-generated NPU code correctness and vectorization efficiency

Innovation

• LLMs generate NPU kernels with compiler feedback
• Open source tools evaluate vectorization efficiency
• Benchmark includes 102 common ML operators