A Finite Time Analysis of Thompson Sampling for Bayesian Optimization with Preferential Feedback

2026-04-27

AI Summary

This work addresses the challenge of Bayesian optimization when only pairwise preference feedback—rather than scalar rewards—is available. It proposes a novel Thompson sampling-based approach that models the latent utility difference via a monotonic link function and introduces a dueling kernel induced by base kernels to handle preference comparisons. The key contributions include the first finite-time regret bound for Thompson sampling under preference feedback, a dual-Thompson-sampling (dual-TS) pairing mechanism, and an anchor-invariance analytical framework. Theoretically, the method is shown to achieve convergence performance comparable to classical Thompson sampling with scalar rewards. Empirical evaluations on both synthetic and real-world datasets demonstrate its effectiveness and validate the theoretical claims.

Abstract

Preference feedback, in the form of pairwise comparisons rather than scalar scores, has seen increasing use in applications such as human-, laboratory-, and expert-in-the-loop design, as well as scientific discovery. We propose a Thompson Sampling (TS) approach to Bayesian optimization with preferential feedback that models comparisons using a monotone link on latent utility differences and leverages the dueling kernel induced by a base kernel. We provide a finite-time analysis showing that the performance of the proposed method matches that of standard TS for conventional Bayesian optimization with scalar feedback. The analysis exploits the anchor invariance of TS for challenger selection and introduces a double-TS pairing variant. We also demonstrate the performance of the method on both synthetic and real-world examples.

Problem

• Thompson Sampling
• Bayesian Optimization
• Preferential Feedback
• Finite Time Analysis
• Pairwise Comparisons

Innovation

• Thompson Sampling
• Preferential Feedback
• Bayesian Optimization
• Dueling Kernel
• Finite-time Analysis