PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning

2026-05-01

AI Summary

This work addresses the challenge that large-scale preference datasets often exhibit inter-annotator disagreement and inconsistency, which hinders robust modeling by a single reward model. To overcome this limitation, the authors propose PrefMoE, a novel mixture-of-experts (MoE) framework for reward learning that introduces trajectory-level soft routing. This architecture adaptively combines multiple experts to capture diverse latent patterns inherent in heterogeneous preferences, while incorporating load-balancing regularization to prevent expert collapse. Experimental results demonstrate that PrefMoE significantly improves the robustness of preference prediction on the D4RL and MetaWorld benchmarks and yields more reliable downstream policy performance, outperforming strong single-model baselines.

Abstract

Preference-based reinforcement learning offers a scalable alternative to manual reward engineering by learning reward structures from comparative feedback. However, large-scale preference datasets, whether collected from crowdsourced annotators or generated by synthetic teachers, often contain heterogeneous and partially conflicting supervision, including disagreement across annotators and inconsistency within annotators. Existing reward learning methods typically fit a single reward model to such data, forcing it to average incompatible signals and thereby limiting robustness. To solve this, we propose PrefMoE, a mixture-of-experts reward learning framework for robust preference modeling. PrefMoE learns multiple specialized reward experts and uses trajectory-level soft routing to combine them adaptively, enabling the model to capture diverse latent preference patterns under noisy and heterogeneous preference supervision. A load-balancing regularizer further stabilizes training by preventing expert collapse. Across locomotion benchmarks from D4RL and manipulation tasks from MetaWorld, PrefMoE improves preference prediction robustness and leads to more reliable downstream policy learning than strong single-model baselines.

Problem

• preference-based reinforcement learning
• heterogeneous supervision
• conflicting preferences
• reward learning
• robustness

Innovation

• Mixture-of-Experts
• Preference-based Reinforcement Learning
• Robust Reward Learning
• Soft Routing
• Load Balancing