Freeform Preference Learning for Robotic Manipulation

2026-06-30

AI Summary

This work addresses the challenge of reward design in long-horizon robotic manipulation tasks, where conventional approaches are limited by sparse success signals and binary human preferences that fail to capture multidimensional aspects of behavioral quality. The authors propose Free-form Preference Learning (FPL), a framework that enables humans to specify arbitrary preference dimensions via natural language and provide multidimensional preference labels over trajectory pairs. FPL leverages these annotations to train a language-conditioned reward model and a reward-conditioned policy, eliminating the need for predefined subtasks or retraining. This approach facilitates dense reward learning, compositional generalization of behaviors, and flexible control at test time. Evaluated on four real-world and two simulated long-horizon tasks, FPL outperforms baselines using sparse rewards or binary preferences by up to 38 percentage points.

Abstract

Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/

Problem

• reward design
• robotic manipulation
• preference learning
• long-horizon tasks
• human preferences

Innovation

• Freeform Preference Learning
• language-conditioned reward
• multi-axis preference
• reward-conditioned policy
• robotic manipulation