Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs

2026-08-27

AI Summary

本文提出MetaNet，通过预测每层的专家保留阈值和路由偏置来动态分配专家，解决了MoE模型中固定专家数量导致的效率问题。

Abstract

Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number of active experts is fixed across layers and tasks, although layer roles and expert redundancy vary with depth and demand varies with difficulty. Existing approaches address only part of this setting: layer-wise allocations are usually determined offline and reused for all tasks, while token-level methods vary expert activation using local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias. The backbone, experts, and router remain frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy-expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) and achieves comparable MMLU accuracy (0.489 vs. 0.474), whereas an aggressive setting activates 2.28 experts on average (62% fewer) with accuracy approximately 3.7 percentage points lower. The MMLU-trained controller also transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.

Problem

• Mixture-of-Experts
• layer-wise allocation
• task-conditioned
• expert activation

Innovation

• MetaNet
• expert-retention threshold
• bounded routing bias
• task-conditioned layer-wise compression