MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE

2025-09-21

AI Summary

To address the low per-token prediction quality of large language models (LLMs) during inference—unamenable to improvement via parameter fine-tuning—this paper proposes RoE, a hyper-parallel inference framework. RoE innovatively repurposes the Mixture-of-Experts (MoE) architecture at inference time as a training-free expert ensemble: it dynamically generates diverse expert proposals via controllable stochastic routing and fuses their outputs. Integrated with multi-proposal aggregation, efficient batched execution, and a dedicated KV cache mechanism, RoE simultaneously reduces computational and memory overhead while enhancing prediction accuracy. Experiments demonstrate that RoE enables a 7B MoE model to outperform a 10.5B dense model in downstream task performance, achieves a 30% reduction in inference FLOPs, and requires zero parameter updates throughout deployment.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The generation quality of large language models (LLMs) is often improved by utilizing inference-time sequence-level scaling methods (e.g., Chain-of-Thought). We introduce hyper-parallel scaling, a complementary framework that improves prediction quality at the token level. Hyper-parallel scaling computes and aggregates multiple output proposals for a single token from the model. We implement this concept in Mixture-of-Experts (MoE) models, which we refer to as Roster of Experts (RoE). RoE is a training-free inference algorithm that turns a single MoE into a dynamic ensemble of MoEs. RoE injects controlled stochasticity into the expert routing mechanism, enabling it to sample multiple diverse experts for each token and aggregate their outputs for a more accurate final prediction.To overcome the computational cost, we introduce an efficient batching strategy and a specialized KV-caching mechanism that minimizes compute and memory overhead. For example, RoE enables a 7B MoE model to match the performance of a 10.5B MoE model while using 30% less compute for inference. These gains are achieved without any fine-tuning of model parameters.

Problem

• Improving token-level prediction quality in large language models
• Reducing computational costs of multiple expert proposals in MoEs
• Enhancing MoE model performance without fine-tuning parameters

Innovation

• Hyper-parallel scaling aggregates multiple token proposals
• RoE algorithm creates dynamic ensemble via stochastic routing
• Efficient batching and KV-caching reduce computational overhead