HeteroLLM: Accelerating Large Language Model Inference on Mobile SoCs platform with Heterogeneous AI Accelerators

2025-01-11

arXiv.org

AI Summary

To address the lack of systematic performance modeling and efficient scheduling for collaborative large language model (LLM) inference across heterogeneous AI accelerators (e.g., GPU, NPU) in mobile SoCs, this paper proposes the first fine-grained heterogeneous execution framework tailored for mobile unified memory architectures. The framework introduces a novel hierarchical and tensor-level dynamic scheduling mechanism that operates separately during the prefill and decoding phases. It incorporates phase-aware dynamic tensor partitioning strategies and lightweight synchronization primitives to fully exploit the unified virtual address space while minimizing cross-device synchronization overhead. Experimental evaluation demonstrates substantial acceleration in mobile LLM inference: our framework achieves 9.99× and 4.36× speedups over MLC and MNN, respectively, establishing it as the fastest known mobile LLM inference engine to date.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

With the rapid advancement of artificial intelligence technologies such as ChatGPT, AI agents and video generation,contemporary mobile systems have begun integrating these AI capabilities on local devices to enhance privacy and reduce response latency. To meet the computational demands of AI tasks, current mobile SoCs are equipped with diverse AI accelerators, including GPUs and Neural Processing Units (NPUs). However, there has not been a comprehensive characterization of these heterogeneous processors, and existing designs typically only leverage a single AI accelerator for LLM inference, leading to suboptimal use of computational resources and memory bandwidth. In this paper, we first summarize key performance characteristics of mobile SoC, including heterogeneous processors, unified memory, synchronization, etc. Drawing on these observations, we propose different tensor partition strategies to fulfill the distinct requirements of the prefill and decoding phases. We further design a fast synchronization mechanism that leverages the unified memory address provided by mobile SoCs. By employing these techniques, we present HeteroLLM, the fastest LLM inference engine in mobile devices which supports both layer-level and tensor-level heterogeneous execution. Evaluation results show that HeteroLLM achieves 9.99 and 4.36 performance improvement over other mobile-side LLM inference engines: MLC and MNN.

Problem

• Characterizing mobile SoC performance for heterogeneous LLM inference
• Optimizing computational resource usage across GPU and NPU accelerators
• Designing synchronization mechanisms for heterogeneous processor collaboration

Innovation

• Characterizing mobile SoC heterogeneous processors performance
• Proposing GPU-NPU parallel mechanisms for computational power
• Designing fast synchronization using unified memory architecture