GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems

2026-05-19

AI Summary

This work addresses the synchronization bottlenecks and inference latency in Mixture-of-Experts (MoE) systems caused by GPU performance heterogeneity and imbalanced expert allocation. To tackle this, the authors propose GEM, a novel framework that explicitly incorporates GPU performance disparities into expert-to-GPU mapping strategies. GEM distinguishes between consistent and transient experts and leverages GPU variability profiling together with task-level token load distribution to dynamically optimize expert placement, aligning computational load with processing capacity. Evaluated against baseline approaches, GEM reduces end-to-end inference latency by 7.9% on average, with improvements reaching up to 16.5%.

Abstract

Mixture-of-Expert (MoE) models enable efficient inference by employing smaller experts and activating only a subset of them per token. MoE serving engines distribute experts across multiple GPUs and route tokens to appropriate GPUs at inference time based on experts activated. They process tokens in lock-step fashion, where tokens within a batch must finish processing before proceeding to the next layer. This synchronization barrier acts as a critical bottleneck because the performance of MoE models is limited by the straggler GPU that finishes last. Stragglers emerge when too many heavily used experts are placed on the same GPU or the slowest GPU. While prior works place experts that balance token loads across GPUs, they all overlook GPU variability and often place highly used experts on the slowest GPUs. We propose GEM, GPU-variability-aware Expert Mapping, a framework for GPU variability-aware expert to GPU mapping for MoE models. GEM exploits two insights. First, we must place experts such that each GPU receives non-uniform token loads based on their variability and they all finish processing a layer at about the same time. Our studies show that there are two types of experts: consistent that are used most of the time and temporal that are often used together for the remaining time. Our second insight is that we must place simultaneously used consistent and temporal experts on different GPUs and avoid placing them on slower GPUs to reduce slowdown. GEM gathers the variability profile of GPUs for each model and task and uses the token load distributions per task to map experts to GPUs. Our experiments show that GEM improves end-to-end latency by 7.9% on average and by up to 16.5% compared to the baseline.

Problem

• Mixture-of-Experts
• GPU variability
• expert placement
• straggler GPU
• inference latency

Innovation

• GPU variability
• expert mapping
• Mixture-of-Experts
• load balancing
• inference optimization