Linear Convergence in Games with Delayed Feedback via Extra Prediction

2026-02-19

AI Summary

Feedback delay is pervasive in multi-agent learning and significantly impedes convergence, yet its precise impact remains unclear—particularly in bilinear games. This work proposes an “extra optimism” mechanism that extends the prediction horizon of the Weighted Optimistic Gradient Descent Ascent (WOGDA) algorithm further into the future to mitigate performance degradation caused by delay. The method can be interpreted as an approximation of the Extrapolated Proximal Point (EPP) algorithm. Theoretically, it accelerates the convergence rate from $\exp(-\Theta(t/m^5))$ under standard optimistic methods to $\exp(-\Theta(t/(m^2 \log m)))$, while also permitting larger step sizes. Experimental results corroborate the theoretical acceleration and demonstrate that the approach overcomes the limitations of conventional optimistic methods in delayed environments.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Mechanism Design

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Feedback delays are inevitable in real-world multi-agent learning. They are known to severely degrade performance, and the convergence rate under delayed feedback is still unclear, even for bilinear games. This paper derives the rate of linear convergence of Weighted Optimistic Gradient Descent-Ascent (WOGDA), which predicts future rewards with extra optimism, in unconstrained bilinear games. To analyze the algorithm, we interpret it as an approximation of the Extra Proximal Point (EPP), which is updated based on farther future rewards than the classical Proximal Point (PP). Our theorems show that standard optimism (predicting the next-step reward) achieves linear convergence to the equilibrium at a rate $\exp(-Θ(t/m^{5}))$ after $t$ iterations for delay $m$. Moreover, employing extra optimism (predicting farther future reward) tolerates a larger step size and significantly accelerates the rate to $\exp(-Θ(t/(m^{2}\log m)))$. Our experiments also show accelerated convergence driven by the extra optimism and are qualitatively consistent with our theorems. In summary, this paper validates that extra optimism is a promising countermeasure against performance degradation caused by feedback delays.

Problem

• delayed feedback
• multi-agent learning
• bilinear games
• convergence rate
• performance degradation

Innovation

• extra optimism
• delayed feedback
• linear convergence
• bilinear games
• optimistic gradient descent-ascent