S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices

2026-08-15

AI Summary

This study addresses the challenges of limited memory bandwidth and excessive verification overhead when deploying Mixture-of-Experts (MoE) large language models on edge devices. We propose S2-MoE, a framework enabling efficient self-speculative decoding through routing-aware adaptive expansion, reuse-aware expert gating, and shared context alignment mechanisms. These components effectively eliminate verification redundancy and enhance expert reuse rates. Experimental evaluations across multiple model datasets demonstrate that S2-MoE achieves an average speedup of approximately 2.0×, with peak acceleration reaching 5.3× on edge hardware. These results indicate significant improvements in MoE inference efficiency for resource-constrained edge environments, offering a practical solution for deploying large-scale sparse models under stringent computational constraints.

Abstract

Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama.cpp, S2-MoE achieves up to 5.3x speedup (about 2.0x on average) over standard autoregressive de?coding across diverse MoE models and datasets on edge devices.Code is available at https://github.com/angerybob/S2-MoE.

Problem

• Edge Devices
• Mixture-of-Experts
• Speculative Decoding
• Inference Efficiency
• Memory Constraints

Innovation

• Self-Speculative Decoding
• Mixture-of-Experts
• Edge Devices
• Routing-Aware Expansion
• Reuse-Aware Gating