ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

2026-07-01

AI Summary

This work addresses the inefficiency in existing decoding-stage routing for MoE large language model serving with prefill-decode separation, which focuses solely on load balancing while ignoring disparities in expert weight loading latency. To tackle this, the authors propose a locality-aware routing mechanism that constructs expert signatures from activation patterns observed during the prefill phase. These signatures are clustered offline using balanced K-means, and incoming requests are routed online to the decoding node with both high signature affinity and low load. By innovatively incorporating expert locality awareness and co-designing indexing for KV cache and signature cache, the method significantly reduces latency while preserving output consistency. Integrated into the vLLM framework and evaluated on 40 GPUs, the approach achieves 5.9%–13.9% lower median time per output token (TPOT) compared to four strong baselines across three MoE models and two workload types.

Abstract

In prefill-decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates. We present ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving. From a request's prefill expert activations, ELDR builds an expert signature predicting the experts it will activate during generation. Offline, balanced K-means partitions signature space across decode workers; online, locality-band routing sends each request to the least-loaded worker among those best matching its signature. A signature cache, co-indexed with the KV cache at KV-block granularity, keeps signatures exact under prefix caching. Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9-13.9% over the strongest of four load-balancing baselines across three MoE models and two workloads, with model outputs unchanged.

Problem

• Mixture-of-Experts
• LLM serving
• decode routing
• expert locality
• PD disaggregation

Innovation

• expert-locality-aware routing
• PD-disaggregated serving
• Mixture-of-Experts (MoE)
• decode routing
• signature-based load balancing