Near-Optimal Last-Iterate Convergence for Zero-Sum Games with Bandit Feedback and Opponent Actions

2026-05-10

AI Summary

This work addresses two-player zero-sum games under bandit feedback, where only the chosen actions' rewards are observed. While prior methods typically achieve faster convergence in averaged iterates than in the last iterate—often limited to rates of $t^{-1/3}$ or $t^{-1/4}$—this paper investigates a setting where the opponent’s actions are observable. The authors propose an efficient algorithm that integrates log-barrier regularized game estimation with sparse strategy updates. Their method establishes, for the first time in this setting, a high-probability last-iterate convergence rate of $t^{-1/2}$, thereby surpassing existing lower bounds and improving upon results in dueling bandits. Both theoretical analysis and empirical experiments confirm the superiority of the proposed approach.

Abstract

Last-iterate convergence of learning dynamics in games has attracted significant recent attention. In two-player zero-sum games with bandit feedback, where only the loss of the selected action pair is observed, Fiegel et al. (2025) show a separation between average-iterate and last-iterate convergence in duality gap: while the optimal t^(-1/2) rate after t rounds is achievable for the former via standard no-regret algorithms, the latter cannot converge faster than t^(-1/3) in expectation or t^(-1/4) with high probability. However, in many practical settings, such as preference learning, the players observe not only their loss but also the opponent's action. This raises a natural question: can such additional information enable faster last-iterate convergence? We answer this question affirmatively, showing that t^(-1/2) last-iterate convergence is achievable with high probability in this setting, via an efficient algorithm that updates its strategy infrequently by solving an estimated log-barrier-regularized game. We identify fundamental obstacles preventing standard analysis for multi-armed bandits, the single-player case, from generalizing to games, and develop a novel analysis to overcome them. Experiments confirm that our algorithm indeed converges faster than naive baselines and prior methods that do not exploit opponent-action feedback. Finally, we note that our results also improve those for dueling bandits, a special case with skew-symmetric game matrices.

Problem

• last-iterate convergence
• zero-sum games
• bandit feedback
• opponent actions
• duality gap

Innovation

• last-iterate convergence
• zero-sum games
• bandit feedback
• opponent-action feedback
• log-barrier regularization