Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare

2026-05-03

AI Summary

This work addresses the fairness challenge in dueling bandits with heterogeneous user preferences, where conventional approaches based on average preferences often disadvantage minority groups. To promote equitable outcomes, the authors propose optimizing Nash social welfare—defined as the product of individual user utilities—using each user’s Condorcet winner as a personalized benchmark. They establish the first quantified fairness cost in this setting by deriving a regret lower bound of Ω(T^{2/3} min(K,D)^{1/3}), where T is the time horizon, K the number of arms, and D the number of user types. Matching this bound, they introduce fair algorithms such as Fair-Explore-Then-Commit and Fair-ε-Greedy, whose regret upper bounds differ only by logarithmic factors, thereby providing strong fairness guarantees across diverse user populations.

Abstract

Learning from human preference data is becoming a useful tool, from fine-tuning large language models to training reinforcement learning agents. However, in most scenarios, the model is trained on the average preference of all human evaluators, which, under large variations of preferences, can be unfair to minority groups. In this work, we consider fairness in dueling bandits, a standard framework for online learning from preference data. We assume that each user has a (potentially distinct) Condorcet winner, which is an arm preferred to every other arm. Using these user-specific Condorcet winners as reference points, we evaluate and score arms according to their performance relative to the corresponding winner. To promote fairness across heterogeneous users, we adopt the well-established Nash Social Welfare objective, which maximizes the product of user utilities, thereby inherently penalizing inequality and preventing the marginalization of any single user. Within this framework, we construct a hard instance to establish a regret lower bound of $Ω(T^{2/3}\min(K,D)^\frac{1}{3})$ for a time horizon $T$, $K$ arms, and $D$ users, which, to the best of our knowledge, is the first result quantifying the cost of fairness in dueling bandits with heterogeneous preferences. We then present the Fair-Explore-Then-Commit and Fair-$ε$-Greedy algorithms with a Condorcet winner identification phase. We further derive their regret upper bounds that match the lower-bound dependence on $T$ up to logarithmic factors.

Problem

• Fairness
• Dueling Bandits
• Heterogeneous Preferences
• Nash Social Welfare
• Multi-User

Innovation

• Multi-User Dueling Bandits
• Nash Social Welfare
• Fairness
• Condorcet Winner
• Regret Bound