Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability

2026-07-13

AI Summary

This work addresses a key limitation in conventional preference-based reinforcement learning: its inability to handle cases where human experts label trajectory pairs as incomparable—neither preferred nor dispreferred. The paper formally introduces incomparability into the framework and proposes a generalized preference-based reinforcement learning approach. Building upon the Bradley–Terry model, it constructs a rational preference model capable of inferring multi-dimensional reward functions from comparison data that includes incomparability annotations. Theoretical analysis demonstrates that the method satisfies rationality conditions and exhibits controllable sample complexity. Empirical results confirm its ability to accurately recover reward functions consistent with expert preferences, effectively reconstruct the policy’s Pareto frontier, and maintain robustness across varying levels of expert rationality.

Abstract

In this work, we study the reinforcement learning (RL) problem from pairwise trajectory comparisons provided by a human expert. We generalize preference-based RL by formalizing a novel setting in which the expert can also label trajectory pairs as incomparable, i.e., when neither trajectory dominates the other. We introduce the learning problem and the desiderata that its solution should satisfy. Then, we propose a novel Bradley-Terry-inspired rationality model that effectively captures incomparabilities and infers a multi-dimensional reward function, and we study its properties. We provide a sample complexity analysis for learning the model parameters when a dataset is available. Finally, we evaluate our model's ability to reconstruct a reward function that aligns with the expert's comparisons in simulated environments and to recover the Pareto frontier of policies, along with a robustness analysis across varying levels of expert rationality.

Problem

• Preference-based Reinforcement Learning
• Incomparability
• Human Feedback
• Multi-dimensional Reward
• Rationality Model

Innovation

• preference-based reinforcement learning
• incomparability
• multi-dimensional reward
• Bradley-Terry model
• Pareto frontier