MoRE: Mixture of Reused Experts

2026-09-16

AI Summary

本文提出MoRE模型，通过在相邻层间共享专家池来解决Mixture-of-Experts架构中参数增加导致内存占用高的问题，同时引入深度嵌入以区分不同层。

Abstract

Mixture-of-Experts (MoE) architectures decouple model capacity from computational cost, yet incur high memory footprints as parameters grow linearly with the number of experts. Recurrent Transformers achieve parameter efficiency by reusing layer weights, but typically lack the capacity for competitive language modeling. We propose Mixture of Reused Experts (MoRE), a hybrid that shares expert pools across groups of adjacent layers. Each layer retains its own router but selects from a larger shared pool, expanding the diversity of routing combinations without additional parameters. To enable shared experts to distinguish between layers, we introduce lightweight learnable depth embeddings that condition each layer's input before routing. Experiments across three model scales (114M-1.15B parameters) show that MoRE consistently achieves lower perplexity and stronger downstream performance than standard MoEs and state-of-the-art weight-sharing architectures at matched compute and parameter budgets, with only minimal modifications to existing MoE implementations.

Problem

• Mixture-of-Experts
• memory footprint
• parameter efficiency
• language modeling

Innovation

• Mixture of Reused Experts
• shared expert pools
• lightweight learnable depth embeddings