Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores

2025-03-13

AI Summary

To address the computational and memory bottlenecks in MoE large language model inference caused by dual redundancy in parameters and activations, this paper proposes the first activation-parameter bilateral structured sparsity co-optimization framework. Methodologically, we design a MoE-specific sparse data format, sparse-sparse GEMM kernels (SpTCs), and a routing-aware sparse scheduling mechanism, systematically adapting the SpTC hardware execution paradigm. Our contributions include: 1.99× kernel-level speedup, 1.58× end-to-end model inference acceleration, and 4.41× increase in batch capacity—all achieved without accuracy loss. Compared to state-of-the-art structured sparsity approaches, our method delivers superior performance while ensuring hardware portability and end-to-end practicality.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

The escalating size of Mixture-of-Experts (MoE) based Large Language Models (LLMs) presents significant computational and memory challenges, necessitating innovative solutions to enhance efficiency without compromising model accuracy. Structured sparsity emerges as a compelling strategy to address these challenges by leveraging the emerging sparse computing hardware. Prior works mainly focus on the sparsity in model parameters, neglecting the inherent sparse patterns in activations. This oversight can lead to additional computational costs associated with activations, potentially resulting in suboptimal performance. This paper presents Samoyeds, an innovative acceleration system for MoE LLMs utilizing Sparse Tensor Cores (SpTCs). Samoyeds is the first to apply sparsity simultaneously to both activations and model parameters. It introduces a bespoke sparse data format tailored for MoE computation and develops a specialized sparse-sparse matrix multiplication kernel. Furthermore, Samoyeds incorporates systematic optimizations specifically designed for the execution of dual-side structured sparse MoE LLMs on SpTCs, further enhancing system performance. Evaluations show that Samoyeds outperforms SOTA works by up to 1.99$ imes$ at the kernel level and 1.58$ imes$ at the model level. Moreover, it enhances memory efficiency, increasing maximum supported batch sizes by 4.41$ imes$ on average. Additionally, Samoyeds surpasses existing SOTA structured sparse solutions in both model accuracy and hardware portability.

Problem

• Addresses computational and memory challenges in large MoE-based LLMs.
• Introduces sparsity in both activations and model parameters for efficiency.
• Enhances performance and memory efficiency using Sparse Tensor Cores.

Innovation

• Applies sparsity to activations and parameters
• Introduces custom sparse data format
• Develops sparse-sparse matrix multiplication kernel