Factor-Aware Mixture-of-Experts with Pretrained Encoder for Combinatorial Generalization

2026-06-19

AI Summary

Existing vision-based robotic manipulation methods exhibit limited generalization under environmental variations such as changes in lighting and texture. This work proposes FAME, a framework that integrates a pretrained visual encoder with a factor-aware dense mixture-of-experts (MoE) architecture. FAME achieves compositional generalization across multiple environmental factors by dynamically weighting frozen, factor-specific lightweight adapters through a central router. The approach employs a three-stage training pipeline: policy pre-warming, factor adapter training, and joint fine-tuning. Evaluated on the Meta-World benchmark, FAME outperforms baseline diffusion policies by 34%; in real-world grasping tasks, it demonstrates a 35% improvement in generalization performance, significantly surpassing current state-of-the-art methods.

Abstract

The integration of pretrained encoders with diffusion policies has become a dominant paradigm for visual robotic manipulation. However, it still struggles to generalize across complex environments with varying factors such as lighting and surface textures. To address this, we propose FAME, a framework that integrates a factor-aware mixture-of-experts (MoE) with a pretrained encoder to enhance generalization to environmental variations. FAME follows a three-stage training process: (1) policy warmup, where a diffusion policy is trained on standard-environment data with a frozen encoder; (2) factor-specific adapter training, where lightweight adapters inserted between the frozen encoder and the temporarily frozen policy are trained on customized datasets, each targeting a distinct environmental variation; and (3) joint fine-tuning, where a central router and the warmed policy are trained on mixed data to handle multiple factors jointly. FAME is ``factor-aware'' because the central router softly weights frozen factor-specific adapters as a dense MoE, enabling combinatorial generalization across multiple factors. Evaluations on the Meta-World benchmark show that FAME outperforms diffusion policy baselines by 34%. We further validate FAME in a real-world pick-and-place task using a compact model trained on newly collected data, where FAME achieves a 35% improvement in generalization under real-world variations.

Problem

• combinatorial generalization
• environmental variations
• visual robotic manipulation
• pretrained encoder
• diffusion policy

Innovation

• Factor-Aware Mixture-of-Experts
• Combinatorial Generalization
• Pretrained Encoder
• Diffusion Policy
• Adapter-based Fine-tuning