FloE: On-the-Fly MoE Inference

2025-05-09

AI Summary

This work addresses the high inference latency of Mixture-of-Experts (MoE) models on memory-constrained devices, caused by excessive expert parameter activation and PCIe bandwidth saturation. We propose a synergistic mechanism combining expert-level fine-grained dynamic compression and low-overhead sparse activation prediction. Our approach integrates parameter block-wise quantization, structured sparse encoding, a lightweight expert activation predictor, and zero-copy CPU-GPU streaming loading—achieving, for the first time, joint optimization of dynamic compression and activation prediction at the expert granularity. On Mixtral-8×7B, our method achieves a per-expert compression ratio of 9.3×, enabling deployment with only 11 GB GPU memory. Compared to DeepSpeed-MII, it improves end-to-end inference throughput by 48.7× and reduces peak GPU memory consumption by up to 8.5%. This work breaks the strong coupling between bandwidth and memory bottlenecks in MoE inference.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Data Mining & Knowledge Management: Data Compression

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

With the widespread adoption of Mixture-of-Experts (MoE) models, there is a growing demand for efficient inference on memory-constrained devices. While offloading expert parameters to CPU memory and loading activated experts on demand has emerged as a potential solution, the large size of activated experts overburdens the limited PCIe bandwidth, hindering the effectiveness in latency-sensitive scenarios. To mitigate this, we propose FloE, an on-the-fly MoE inference system on memory-constrained GPUs. FloE is built on the insight that there exists substantial untapped redundancy within sparsely activated experts. It employs various compression techniques on the expert's internal parameter matrices to reduce the data movement load, combined with low-cost sparse prediction, achieving perceptible inference acceleration in wall-clock time on resource-constrained devices. Empirically, FloE achieves a 9.3x compression of parameters per expert in Mixtral-8x7B; enables deployment on a GPU with only 11GB VRAM, reducing the memory footprint by up to 8.5x; and delivers a 48.7x inference speedup compared to DeepSpeed-MII on a single GeForce RTX 3090.

Problem

• Efficient MoE inference on memory-constrained devices
• Reducing PCIe bandwidth overload from large expert activations
• Compressing expert parameters to accelerate resource-limited inference

Innovation

• Compresses expert parameters to reduce data movement
• Enables MoE inference on memory-constrained GPUs
• Combines compression with low-cost sparse prediction