Bandits with Preference Feedback: A Stackelberg Game Perspective

2024-06-24

Neural Information Processing Systems

AI Summary

This paper studies preference-based bandit optimization with nonlinear (kernelized) reward functions over infinite action spaces, where only pairwise action comparisons are available as relative feedback, and both intra-action and inter-iteration exploration-exploitation trade-offs must be addressed. We formulate the problem for the first time as a zero-sum Stackelberg game. We propose MAXMINLCB, a novel algorithm that constructs preference-based kernelized logistic regression confidence sequences, yielding an anytime-valid, rate-optimal regret bound of $O(sqrt{gamma_T T})$, where $gamma_T$ denotes the information gain in the associated reproducing kernel Hilbert space. Compared to prior approaches, our method achieves significant advances both theoretically—by establishing the first rate-optimal, anytime-valid regret guarantee under general kernelized preferences—and empirically—by demonstrating superior practical performance across benchmarks.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Bandits with preference feedback present a powerful tool for optimizing unknown target functions when only pairwise comparisons are allowed instead of direct value queries. This model allows for incorporating human feedback into online inference and optimization and has been employed in systems for fine-tuning large language models. The problem is well understood in simplified settings with linear target functions or over finite small domains that limit practical interest. Taking the next step, we consider infinite domains and nonlinear (kernelized) rewards. In this setting, selecting a pair of actions is quite challenging and requires balancing exploration and exploitation at two levels: within the pair, and along the iterations of the algorithm. We propose MAXMINLCB, which emulates this trade-off as a zero-sum Stackelberg game, and chooses action pairs that are informative and yield favorable rewards. MAXMINLCB consistently outperforms existing algorithms and satisfies an anytime-valid rate-optimal regret guarantee. This is due to our novel preference-based confidence sequences for kernelized logistic estimators.

Problem

• Optimizes unknown functions using pairwise comparisons
• Handles infinite domains and nonlinear kernelized rewards
• Balances exploration-exploitation via a Stackelberg game approach

Innovation

• Emulates trade-off as zero-sum Stackelberg game
• Uses novel preference-based confidence sequences
• Applies to infinite domains with nonlinear rewards