Pessimism-Free Offline Learning in General-Sum Games via KL Regularization

2026-04-30

AI Summary

This work addresses the instability of policies in offline multi-agent reinforcement learning under general-sum games, which arises from distributional shift in the logged data. The authors propose a novel approach that eschews pessimistic assumptions by replacing conventional pessimistic penalties with KL regularization. Their method integrates anchored Nash equilibrium modeling with a mirror descent algorithm (GAMD) for policy optimization. Theoretical analysis demonstrates, for the first time, that KL regularization alone can ensure stable learning: the proposed GANE algorithm recovers a regularized Nash equilibrium at a rate of Õ(1/n), while GAMD converges to a coarse correlated equilibrium at a rate of Õ(1/√n + 1/T), substantially improving both convergence speed and robustness.

Abstract

Offline multi-agent reinforcement learning in general-sum settings is challenged by the distribution shift between logged datasets and target equilibrium policies. While standard methods rely on manual pessimistic penalties, we demonstrate that KL regularization suffices to stabilize learning and achieve equilibrium recovery. We propose General-sum Anchored Nash Equilibrium (GANE), which recovers regularized Nash equilibria at an accelerated statistical rate of $\widetilde{O}(1/n)$. For computational tractability, we develop General-sum Anchored Mirror Descent (GAMD), an iterative algorithm converging to a Coarse Correlated Equilibrium at the standard rate of $\widetilde{O}(1/\sqrt{n}+1/T)$. These results establish KL regularization as a standalone mechanism for pessimism-free offline learning that achieves equivalent or accelerated rates in multi-player general-sum games.

Problem

• offline reinforcement learning
• general-sum games
• distribution shift
• multi-agent learning
• equilibrium recovery

Innovation

• KL regularization
• offline multi-agent reinforcement learning
• general-sum games
• Nash equilibrium recovery
• distribution shift