From Demonstrations to Rewards: Alignment Without Explicit Human Preferences

2025-03-15

AI Summary

To address the high cost and complexity of human preference annotation in large language model (LLM) alignment, this paper proposes a novel inverse reinforcement learning (IRL) paradigm that relies solely on demonstration data. We theoretically establish that demonstration data implicitly encodes human preferences, thereby eliminating the need for explicit preference labeling in conventional RLHF and enabling end-to-end alignment. Methodologically, we introduce a joint optimization framework that simultaneously trains the policy and infers the reward function, ensuring compatibility with mainstream LLM architectures and open-source evaluation benchmarks. Experiments demonstrate that our approach achieves performance on par with or superior to current state-of-the-art demonstration-only methods across the Hugging Face Open LLM Leaderboard, MT-Bench, and public reward modeling benchmarks—while substantially reducing both data curation effort and engineering overhead.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

One of the challenges of aligning large models with human preferences lies in both the data requirements and the technical complexities of current approaches. Predominant methods, such as RLHF, involve multiple steps, each demanding distinct types of data, including demonstration data and preference data. In RLHF, human preferences are typically modeled through a reward model, which serves as a proxy to guide policy learning during the reinforcement learning stage, ultimately producing a policy aligned with human preferences. However, in this paper, we propose a fresh perspective on learning alignment based on inverse reinforcement learning principles, where the optimal policy is still derived from reward maximization. However, instead of relying on preference data, we directly learn the reward model from demonstration data. This new formulation offers the flexibility to be applied even when only demonstration data is available, a capability that current RLHF methods lack, and it also shows that demonstration data offers more utility than what conventional wisdom suggests. Our extensive evaluation, based on public reward benchmark, HuggingFace Open LLM Leaderboard and MT-Bench, demonstrates that our approach compares favorably to state-of-the-art methods that rely solely on demonstration data.

Problem

• Aligning large models with human preferences efficiently
• Reducing reliance on explicit human preference data
• Learning reward models directly from demonstration data

Innovation

• Uses inverse reinforcement learning principles
• Learns reward model from demonstration data
• Applies without needing explicit preference data