Reviving The Classics: Active Reward Modeling in Large Language Model Alignment

2025-02-04

AI Summary

To address the inefficiency of reward modeling in RLHF caused by scarce and costly human preference annotations, this paper proposes a novel active reward modeling framework. Our method is the first to integrate the Fisher Information Criterion with optimal experimental design theory into LLM reward modeling, jointly optimizing representation-space explorability and discriminability of moderate reward differences to select high-information preference pairs. We further introduce a cross-prompt contrastive mechanism and construct an efficient active learning framework grounded in linear-layer sensitivity analysis. Extensive experiments across multiple open-source LLMs and benchmark datasets demonstrate that our approach significantly outperforms existing active learning baselines, achieving superior trade-offs among annotation efficiency, model performance, training stability, and computational overhead.

Technology Category

• Machine Learning: Active Learning
• Search and Optimization: Learning to Search
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Building neural reward models from human preferences is a pivotal component in reinforcement learning from human feedback (RLHF) and large language model alignment research. Given the scarcity and high cost of human annotation, how to select the most informative pairs to annotate is an essential yet challenging open problem. In this work, we highlight the insight that an ideal comparison dataset for reward modeling should balance exploration of the representation space and make informative comparisons between pairs with moderate reward differences. Technically, challenges arise in quantifying the two objectives and efficiently prioritizing the comparisons to be annotated. To address this, we propose the Fisher information-based selection strategies, adapt theories from the classical experimental design literature, and apply them to the final linear layer of the deep neural network-based reward modeling tasks. Empirically, our method demonstrates remarkable performance, high computational efficiency, and stability compared to other selection methods from deep learning and classical statistical literature across multiple open-source LLMs and datasets. Further ablation studies reveal that incorporating cross-prompt comparisons in active reward modeling significantly enhances labeling efficiency, shedding light on the potential for improved annotation strategies in RLHF.

Problem

• active reward modeling
• human preference annotation
• Fisher information selection

Innovation

• Fisher information-based selection strategies
• Adapts classical experimental design theories
• Enhances labeling efficiency with cross-prompt comparisons