Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control

2026-07-04

AI Summary

Existing world models lack a unified and scalable framework capable of integrating heterogeneous action control signals while preserving shared dynamics modeling. This work proposes a diffusion Transformer–based Mixture-of-Experts (MoE) world model that, for the first time, unifies multiple action interfaces within a single architecture through modality-aware control injection and a hybrid structure of shared and specialized experts. We introduce a progressive MoE fine-tuning strategy that enables continual integration of new action modalities without interfering with previously acquired capabilities. Evaluated on the WorldArena benchmark, the model demonstrates significant improvements in locomotion and dexterous manipulation, exhibits strong out-of-distribution generalization, and shows clear scalability with increasing numbers of experts and training data volume.

Abstract

World models are rapidly becoming a core infrastructure for embodied intelligence and interactive agents: they provide controllable simulators in which agents can perceive, act, forecast, and acquire scalable experience. Yet current video generation world models are still organized around isolated control interfaces, such as camera trajectories, robot actions, or hand-joint signals. This fragmentation is increasingly a scaling bottleneck. The central challenge is not the absence of controllable generators, but the lack of a unified and extensible learning framework that can absorb heterogeneous action supervision while preserving a shared model of world dynamics. In this work, we introduce Worldscape-MoE, a Mixture-of-Experts world model built on Diffusion Transformers for scalable heterogeneous action control. Our key observation is that different controls specify different interfaces to the same underlying world: although their representations differ, they constrain shared physical regularities, scene dynamics, and interaction semantics. Worldscape-MoE operationalizes this observation through modality-aware control injection, shared and control-specific experts, and a progressive MoE tuning strategy that supports continual extension to new action modalities. Experiments across locomotion, robotic manipulation, and egocentric hand control show that heterogeneous supervision improves rather than interferes with individual control capabilities. Worldscape-MoE achieves strong results on WorldArena, improves locomotion and hand-control metrics, exhibits robust out-of-distribution generalization, and demonstrates scaling behavior as additional control data and experts are integrated.

Problem

• world models
• heterogeneous action control
• scalable learning framework
• unified control interface
• embodied intelligence

Innovation

• Mixture-of-Experts
• World Model
• Heterogeneous Action Control
• Diffusion Transformer
• Scalable Embodied Intelligence