Semantically Structured Mixture-of-Experts for Compositional Robotic Manipulation

2026-05-22

AI Summary

This work addresses the high computational cost of diffusion-based policies in multitask robotic manipulation, the poor generalization of lightweight models, and the behavioral fragmentation caused by existing Mixture-of-Experts (MoE) routing strategies that neglect task structure. To overcome these limitations, the authors propose SMoDP, a semantically structured MoE framework that aligns expert modules with language-defined task phases. SMoDP incorporates dual contrastive learning—both cross-modal and intra-modal—to enhance semantic consistency and routing stability. By integrating a vision-language model–supervised skill predictor with a chunk-wise action routing mechanism, SMoDP achieves superior performance over current diffusion and MoE approaches while maintaining parameter efficiency. Furthermore, it enables compositional transfer to novel tasks through parameter-efficient fine-tuning.

Abstract

Diffusion-based policies have established a new standard for precise robotic manipulation but face a critical scalability bottleneck: high-performance models are computationally expensive, while lightweight alternatives often fail to generalize across diverse multi-task environments. Mixture-of-Experts (MoE) architectures offer a promising path to efficiency by activating only a subset of parameters. However, existing MoE routing mechanisms typically rely on low-level noise or latent statistics, ignoring the compositional nature of manipulation tasks. This can fragment reusable behaviors across experts, limiting interpretability and transferability. We introduce Semantically Structured Mixture-of-Experts Diffusion Policy (SMoDP) for compositional robotic manipulation, a framework that grounds expert specialization in semantic task structure. SMoDP leverages a lightweight, inference-time skill predictor, supervised by offline annotations from Vision-Language Models (VLMs), to route action chunks to experts specialized for specific behavioral phases. To ensure robust assignment, we propose a dual contrastive alignment strategy that grounds multi-modal observations in language-defined skill semantics (Inter-modal) while enforcing routing consistency across visually distinct but functionally related behaviors (Intra-modal). Our approach outperforms representative diffusion and MoE-based baselines on multi-task benchmarks with significantly improved parameter efficiency and demonstrates effective compositional transfer to novel tasks through parameter-efficient fine-tuning. Project website: https://deng-cy20.github.io/SMoDP/

Problem

• robotic manipulation
• Mixture-of-Experts
• compositional tasks
• scalability bottleneck
• generalization

Innovation

• Mixture-of-Experts
• Semantic Routing
• Diffusion Policy
• Compositional Manipulation
• Vision-Language Models