Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models

2026-06-15

AI Summary

This work addresses the substantial memory overhead in Mixture-of-Experts (MoE) language models, which arises from loading all expert parameters during both training and inference. The authors propose a cross-layer expert parameter tying strategy that shares expert weights across consecutive Transformer layers while preserving layer-specific routing mechanisms and attention architectures. Evaluated on prominent MoE frameworks—including OLMoE, Qwen3, and DeepSeek—the approach achieves nearly a 2× reduction in memory consumption with negligible degradation in perplexity or downstream task performance. This advancement significantly improves the trade-off between computational efficiency and memory usage, enabling more scalable deployment of large MoE models without compromising model expressivity or task accuracy.

Abstract

Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-memory trade-off, advancing efficient training and scaling of next-generation LLMs.

Problem

• Mixture-of-Experts
• Large Language Models
• memory footprint
• parameter redundancy
• efficient scaling

Innovation

• Expert Tying
• Mixture-of-Experts
• Memory Efficiency
• Parameter Sharing
• Large Language Models