A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems

2026-01-07

arXiv.org

AI Summary

This work addresses three key challenges in MoE model inference on edge GPU–NDP systems: imbalanced NDP workload, underutilized GPU resources, and high pre-analysis overhead caused by unpredictable expert activation. To tackle these issues, the paper introduces tensor parallelism for the first time in low-batch edge scenarios, integrating it with a load-aware scheduling strategy and a pre-analysis-free expert prefetching mechanism to form a cooperative GPU–NDP computing architecture. The proposed approach significantly enhances resource utilization and inference efficiency, achieving an average end-to-end latency speedup of 2.41× and up to 2.56× over existing methods, demonstrating clear superiority in edge deployment settings.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Mixture-of-Experts (MoE) models facilitate edge deployment by decoupling model capacity from active computation, yet their large memory footprint drives the need for GPU systems with near-data processing (NDP) capabilities that offload experts to dedicated processing units. However, deploying MoE models on such edge-based GPU-NDP systems faces three critical challenges: 1) severe load imbalance across NDP units due to non-uniform expert selection and expert parallelism, 2) insufficient GPU utilization during expert computation within NDP units, and 3) extensive data pre-profiling necessitated by unpredictable expert activation patterns for pre-fetching. To address these challenges, this paper proposes an efficient inference framework featuring three key optimizations. First, the underexplored tensor parallelism in MoE inference is exploited to partition and compute large expert parameters across multiple NDP units simultaneously towards edge low-batch scenarios. Second, a load-balancing-aware scheduling algorithm distributes expert computations across NDP units and GPU to maximize resource utilization. Third, a dataset-free pre-fetching strategy proactively loads frequently accessed experts to minimize activation delays. Experimental results show that our framework enables GPU-NDP systems to achieve 2.41x on average and up to 2.56x speedup in end-to-end latency compared to state-of-the-art approaches, significantly enhancing MoE inference efficiency in resource-constrained environments.

Problem

• Mixture-of-Experts
• Edge Computing
• Near-Data Processing
• Load Imbalance
• GPU Utilization

Innovation

• Mixture-of-Experts
• Near-Data Processing
• Tensor Parallelism
• Load-Balancing Scheduling
• Dataset-Free Prefetching