HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference

2026-07-14

AI Summary

This work addresses the limitations of existing edge LLM inference runtimes on heterogeneous SoCs, which typically support only coarse-grained device selection or isolated operator optimization, thereby failing to jointly exploit CPUs, integrated GPUs (iGPUs), and NPUs—especially under unified memory architectures where joint optimization of task graphs and scheduling is lacking. To overcome this, we propose a heterogeneity-aware scheduling framework that leverages a heterogeneous Roofline model to evaluate the collaborative benefits of iGPUs and NPUs, decomposes inference into dependency-preserving micro-batches for overlapping execution across accelerators, and jointly optimizes scheduling and resource allocation by incorporating real-world constraints such as memory contention, DVFS, and device-specific characteristics. Experiments on the AMD Ryzen AI platform demonstrate up to 2.35× speedup over state-of-the-art heterogeneous approaches, 2.05× improvement over llama.cpp, and a 45.3% reduction in energy consumption.

Abstract

Modern edge system-on-chips (SoCs) combine CPUs, integrated GPUs (iGPUs), and neural processing units (NPUs), yet existing LLM runtimes typically make coarse device-level decisions or optimize operators in isolation. As a result, they underutilize heterogeneous resources, particularly on unified-memory platforms where performance depends on both device placement and task-graph coordination. We present HeteroMosaic, a heterogeneity-first scheduling framework for edge LLM inference. HeteroMosaic first uses a heterogeneous roofline model to identify when combining iGPU and NPU execution is beneficial. It then decomposes inference into dependency-preserving micro-batches that expose cross-accelerator overlap and applies trace-guided co-optimization of scheduling and device allocation under practical effects such as memory contention, DVFS, device variation, and NPU runtime overheads. We implement HeteroMosaic in PyTorch C++ and evaluate it on three AMD Ryzen AI platforms spanning NPU-heavy, balanced, and iGPU-heavy designs. On the balanced platform, HeteroMosaic achieves up to 1.73X speedup over an iGPU baseline, 1.78X over an NPU baseline, and 2.05X over frameworks such as \texttt{llama.cpp}, while reducing energy by up to 45.3%. It also improves performance over prior heterogeneous edge AI solutions by up to 2.35X.

Problem

• heterogeneous computing
• edge LLM inference
• unified memory
• device placement
• task-graph coordination

Innovation

• heterogeneous scheduling
• edge LLM inference
• micro-batching
• roofline model
• cross-accelerator overlap