Knowledge Gradient for Preference Learning

2026-01-29

AI Summary

This work addresses the challenge in preference-based Bayesian optimization, where only pairwise comparison feedback is available, rendering traditional knowledge gradient methods inapplicable due to non-Gaussian posteriors. The authors propose the first analytically tractable knowledge gradient acquisition function tailored to this setting. By modeling the latent utility with a Gaussian process and incorporating a Bayesian inference framework that accounts for preference observations, the method enables efficient posterior updates and optimization. It overcomes the computational intractability associated with non-Gaussian posteriors and demonstrates significant performance gains over existing acquisition functions across multiple benchmark problems. Case studies further elucidate the method’s practical applicability and limitations.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

The knowledge gradient is a popular acquisition function in Bayesian optimization (BO) for optimizing black-box objectives with noisy function evaluations. Many practical settings, however, allow only pairwise comparison queries, yielding a preferential BO problem where direct function evaluations are unavailable. Extending the knowledge gradient to preferential BO is hindered by its computational challenge. At its core, the look-ahead step in the preferential setting requires computing a non-Gaussian posterior, which was previously considered intractable. In this paper, we address this challenge by deriving an exact and analytical knowledge gradient for preferential BO. We show that the exact knowledge gradient performs strongly on a suite of benchmark problems, often outperforming existing acquisition functions. In addition, we also present a case study illustrating the limitation of the knowledge gradient in certain scenarios.

Problem

• knowledge gradient
• preference learning
• Bayesian optimization
• non-Gaussian posterior
• pairwise comparisons

Innovation

• Knowledge Gradient
• Preferential Bayesian Optimization
• Non-Gaussian Posterior
• Acquisition Function
• Pairwise Comparisons