Group Distributionally Robust Optimization with Flexible Sample Queries

2025-05-21

AI Summary

Existing group distributionally robust optimization (GDRO) methods assume fixed batch sizes, rendering them unsuitable for real-world settings with dynamically varying sample counts per group. This work formulates dynamic-batch GDRO as a two-player game between online convex optimization and prediction-with-limited-advice (PLA), the first such modeling. We propose the first non-blind PLA algorithm with a high-probability regret bound. Building upon this, we design multiple GDRO algorithms that accommodate arbitrary, round-wise variable group sample sizes while preserving optimal sample complexity $O(m log m / varepsilon^2)$. Leveraging adaptive loss estimation, Follow-the-Regularized-Leader updates, and high-probability analysis, we derive an optimization error bound of $Oig((1/t)sqrt{sum_j (m/r_j) log m}ig)$, where $r_j$ denotes the number of samples from group $j$. Extensive experiments on synthetic binary classification and real-world multiclass benchmarks validate the efficacy and robustness of our approach.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Group distributionally robust optimization (GDRO) aims to develop models that perform well across $m$ distributions simultaneously. Existing GDRO algorithms can only process a fixed number of samples per iteration, either 1 or $m$, and therefore can not support scenarios where the sample size varies dynamically. To address this limitation, we investigate GDRO with flexible sample queries and cast it as a two-player game: one player solves an online convex optimization problem, while the other tackles a prediction with limited advice (PLA) problem. Within such a game, we propose a novel PLA algorithm, constructing appropriate loss estimators for cases where the sample size is either 1 or not, and updating the decision using follow-the-regularized-leader. Then, we establish the first high-probability regret bound for non-oblivious PLA. Building upon the above approach, we develop a GDRO algorithm that allows an arbitrary and varying sample size per round, achieving a high-probability optimization error bound of $Oleft(frac{1}{t}sqrt{sum_{j=1}^t frac{m}{r_j}log m} ight)$, where $r_t$ denotes the sample size at round $t$. This result demonstrates that the optimization error decreases as the number of samples increases and implies a consistent sample complexity of $O(mlog (m)/epsilon^2)$ for any fixed sample size $rin[m]$, aligning with existing bounds for cases of $r=1$ or $m$. We validate our approach on synthetic binary and real-world multi-class datasets.

Problem

• Enables flexible sample sizes in GDRO algorithms
• Formulates GDRO as two-player game for optimization
• Achieves high-probability error bounds with varying samples

Innovation

• Flexible sample queries in GDRO
• Two-player game optimization approach
• Novel PLA algorithm with loss estimators