Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences

2026-07-14

AI Summary

Existing robotic policy learning approaches struggle to accommodate the diversity of user preferences: single-policy methods overlook preference heterogeneity, while user-specific alignment suffers from sparse and noisy feedback as well as high validation costs. To address these limitations, this work proposes the PREC framework, which jointly optimizes user preference clustering and reward learning. PREC first employs a shared trajectory encoder to extract universal representations, then simultaneously performs user clustering and cluster-wise reward modeling using preference labels. This enables the optimization of a dedicated policy for each cluster, preserving preference diversity while mitigating label sparsity and noise, and substantially reducing pre-deployment validation burden. Experiments in simulated locomotion environments demonstrate that PREC more accurately identifies groups of users with consistent preferences and yields policies that outperform single-policy baselines—and even surpass user-specific alignment—across three social welfare metrics.

Abstract

Aligning robot policies with human preferences is essential for deployment to diverse end users. In per-user alignment approach, preference feedback is often sparse, so learning becomes unstable and vulnerable to human preference noise, and a growing number of individualized policies makes validation difficult before deployment. A single shared policy approach to user alignment avoids this cost but fails to capture heterogeneous preferences and often neglects minority preferences. To address these challenges, we introduce Preference-based REward Clustering (PREC), a novel framework that learns a compact set of policies from binary preference labels provided by diverse users. From a dataset of user trajectories and their preference labels, PREC first sets the labels aside and aggregates trajectories across users to learn a population-level shared trajectory encoder, alleviating limited per-user coverage and avoiding label noise during representation learning. Using this representation, PREC jointly assigns users to preference-coherent clusters and learns a representative reward model per cluster using preference labels, from which a policy is optimized for each cluster. Clustering similar users compensates for the limited number of labels available from each user and mitigates the effect of label noise. At the same time, maintaining a manageable number of reward models reduces the validation burden at deployment. Experiments across diverse simulated locomotion environments show that PREC groups users who label different trajectory subsets into preference-coherent clusters more accurately than baseline methods. Under sparse and noisy feedback, policies trained with PREC improve all three social welfare metrics over an existing single shared-policy user-alignment approach and even outperform per-user alignment approaches.

Problem

• human preference alignment
• robotics
• reward learning
• preference heterogeneity
• deployable AI

Innovation

• Preference-based Reward Clustering
• Human Preference Alignment
• Reward Modeling
• Policy Clustering
• Shared Trajectory Encoder