The Harder Path: Last Iterate Convergence for Uncoupled Learning in Zero-Sum Games with Bandit Feedback

2026-04-17

AI Summary

This work addresses whether decoupled learning algorithms—requiring no communication between players—can guarantee convergence of the last-iterate strategies to a Nash equilibrium in repeated two-player zero-sum matrix games with bandit feedback. To tackle this question, the authors propose two regularized algorithms based on two-step mirror descent and an exploration–exploitation trade-off mechanism. Under the fully decoupled setting with only bandit feedback, they establish, for the first time, a theoretical lower bound of Ω(T⁻¹/⁴) on the convergence rate of last-iterate exploitability. Furthermore, they design an algorithm that matches this lower bound, achieving an optimal convergence rate of O(T⁻¹/⁴), up to constant and logarithmic factors.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Mechanism Design
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Economics, Online Markets and Human Computation: Social networks and social learning
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We study the problem of learning in zero-sum matrix games with repeated play and bandit feedback. Specifically, we focus on developing uncoupled algorithms that guarantee, without communication between players, the convergence of the last-iterate to a Nash equilibrium. Although the non-bandit case has been studied extensively, this setting has only been explored recently, with a bound of $\mathcal{O}(T^{-1/8})$ on the exploitability gap. We show that, for uncoupled algorithms, guaranteeing convergence of the policy profiles to a Nash equilibrium is detrimental to the performance, with the best attainable rate being $Ω(T^{-1/4})$ in contrast to the usual $Ω(T^{-1/2})$ rate for convergence of the average iterates. We then propose two algorithms that achieve this optimal rate up to constant and logarithmic factors. The first algorithm leverages a straightforward trade-off between exploration and exploitation, while the second employs a regularization technique based on a two-step mirror descent approach.

Problem

• zero-sum games
• bandit feedback
• uncoupled learning
• last-iterate convergence
• Nash equilibrium

Innovation

• last-iterate convergence
• bandit feedback
• uncoupled learning
• zero-sum games
• Nash equilibrium