ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving

2026-04-16

AI Summary

This work addresses the challenges of deploying Mixture-of-Experts (MoE) models locally, where large batch sizes lead to memory-intensive workloads and low computational utilization, and conventional speculative decoding fails to mitigate expert-loading overhead. The authors propose ELMoE-3D, a hardware-software co-design framework leveraging 3D hybrid bonding, which for the first time reveals dual elasticity in MoE models across both expert and bit dimensions. Central to this framework is Elastic Self-Speculative Decoding (Elastic-SD), integrating expert caching with a strongly aligned self-draft model and an LSB-enhanced bit-slicing architecture to enable native bit-level nested execution. Evaluated across batch sizes of 1–16, ELMoE-3D achieves an average 6.6× speedup and 4.4× higher energy efficiency over general-purpose xPUs, and further delivers 2.2× acceleration and 1.4× better energy efficiency compared to the state-of-the-art MoE accelerators.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Mixed Discrete/Continuous Search
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Mixture-of-Experts (MoE) models have become the dominant architecture for large-scale language models, yet on-premises serving remains fundamentally memory-bound as batching turns sparse per-token compute into dense memory activation. Memory-centric architectures (PIM, NMP) improve bandwidth but leave compute underutilized under MoE's low arithmetic intensity at high batch sizes. Speculative decoding (SD) trades idle compute for fewer target invocations, yet verification must load experts even for rejected tokens, severely limiting its benefit in MoE especially at low batch sizes. We propose ELMoE-3D, a hybrid-bonding (HB)-based HW-SW co-designed framework that unifies cache-based acceleration and speculative decoding to offer overall speedup across batch sizes. We identify two intrinsic elasticity axes of MoE-expert and bit-and jointly scale them to construct Elastic Self-Speculative Decoding (Elastic-SD), which serves as both an expert cache and a strongly aligned self-draft model accelerated by high HB bandwidth. Our LSB-augmented bit-sliced architecture exploits inherent redundancy in bit-slice representations to natively support bit-nested execution. On our 3D-stacked hardware, ELMoE-3D achieves an average $6.6\times$ speedup and $4.4\times$ energy efficiency gain over naive MoE serving on xPU across batch sizes 1-16, and delivers $2.2\times$ speedup and $1.4\times$ energy efficiency gain over the best-performing prior accelerator baseline.

Problem

• Mixture-of-Experts
• on-premises serving
• memory-bound
• speculative decoding
• batch size

Innovation

• Mixture-of-Experts
• Speculative Decoding
• Hybrid Bonding
• Elasticity
• 3D-Stacked Architecture