MAPL: Multi-Objective Preference Learning for Robot Locomotion

2026-06-24

AI Summary

This work addresses the challenges of manually designing reward functions for robotic motion control and the limitations of existing preference learning methods in modeling trade-offs among multiple objectives. To overcome these issues, the authors propose a large language model (LLM)-based multi-objective preference learning approach that leverages natural language descriptions to independently compare trajectories across semantically meaningful objective dimensions. A multi-headed preference scoring model is constructed from these comparisons and aggregated into a scalar reward signal for policy optimization. The method eliminates the need for task-specific reward engineering or domain expertise, enabling terrain-agnostic policy learning. Evaluated on a quadruped robot across four complex environments, policies trained solely with LLM-generated preferences achieve performance comparable to or exceeding that of methods using expert-designed rewards.

Abstract

Reward design remains a major bottleneck in reinforcement learning for robot locomotion, where successful policies often depend on carefully tuned, task-specific reward functions. Preference-based reinforcement learning offers an alternative, but existing LLM-based methods typically ask for a single overall judgment between behaviors, making it difficult to capture the multiple competing objectives that underlie high-quality locomotion. We present Multi-Objective AI-Informed Preference Learning (MAPL), a framework that learns locomotion rewards from high-level natural language objectives rather than manually engineered reward equations. MAPL prompts a large language model to compare trajectories independently along semantically meaningful criteria, using generic language descriptions that are terrain-invariant and require little domain expertise. These objective-wise preferences are used to train a multi-head preference scoring model, whose outputs are aggregated to form a scalar reward for policy optimization. Across four quadruped locomotion environments, MAPL trains policies using only LLM-generated preferences and achieves performance comparable to or better than expert-designed rewards, while eliminating task-specific reward engineering.

Problem

• reward design
• robot locomotion
• multi-objective preference
• reinforcement learning
• preference-based learning

Innovation

• Multi-Objective Preference Learning
• Large Language Model
• Reward-Free Reinforcement Learning
• Robot Locomotion
• Natural Language Objectives