Consecutive Preferential Bayesian Optimization

2025-11-07

AI Summary

For costly black-box optimization problems requiring human expert pairwise comparisons, existing preference-based Bayesian optimization (PBO) methods neglect both the computational cost of candidate solution generation and the perceptual uncertainty inherent in human evaluators—specifically, the just-noticeable difference (JND) threshold. This paper introduces a continuous-preference Bayesian optimization framework that, for the first time, jointly and explicitly models generation cost and perceptual uncertainty. We propose an information-driven sampling strategy that integrates the JND threshold with expected information gain, grounded in a probabilistic preference model. The method operates under realistic constraints, including indifference intervals in feedback and high generation costs, significantly improving convergence accuracy and sample efficiency. Experiments demonstrate that, under identical evaluation budgets, our approach achieves an average 23.6% improvement in accuracy over baseline PBO methods.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Preferential Bayesian optimization allows optimization of objectives that are either expensive or difficult to measure directly, by relying on a minimal number of comparative evaluations done by a human expert. Generating candidate solutions for evaluation is also often expensive, but this cost is ignored by existing methods. We generalize preference-based optimization to explicitly account for production and evaluation costs with Consecutive Preferential Bayesian Optimization, reducing production cost by constraining comparisons to involve previously generated candidates. We also account for the perceptual ambiguity of the oracle providing the feedback by incorporating a Just-Noticeable Difference threshold into a probabilistic preference model to capture indifference to small utility differences. We adapt an information-theoretic acquisition strategy to this setting, selecting new configurations that are most informative about the unknown optimum under a preference model accounting for the perceptual ambiguity. We empirically demonstrate a notable increase in accuracy in setups with high production costs or with indifference feedback.

Problem

• Optimizes expensive objectives using human comparative evaluations
• Reduces production costs by reusing previously generated candidates
• Accounts for perceptual ambiguity in human feedback using thresholds

Innovation

• Constrains comparisons to previously generated candidates
• Incorporates Just-Noticeable Difference threshold model
• Uses information-theoretic acquisition for optimal configurations