🤖 AI Summary
This paper studies zero-sum networked Markov games (zero-sum NMGs), a novel class of multi-player zero-sum Markov games wherein state-dependent auxiliary games exhibit polymatrix (neighborhood-separable) structure, modeling local network interactions in non-cooperative multi-agent sequential decision-making.
Method: We establish the first necessary and sufficient characterization of zero-sum NMGs; prove equivalence between Markov coarse correlated equilibria (CCE) and Markov Nash equilibria (NE); identify star-topology networks as a key structural condition that circumvents PPAD-hardness and ensures convergence; and design network-adapted fictitious play dynamics and non-stationary value iteration algorithms.
Results: We theoretically prove that fictitious play converges to stationary Markov NE in star-structured zero-sum NMGs, and that non-stationary Markov NE can be computed exactly in finite steps via value iteration, with explicit convergence bounds. Numerical experiments validate the theoretical findings.
📝 Abstract
We study a new class of Markov games, emph(multi-player) zero-sum Markov Games} with emph{Networked separable interactions} (zero-sum NMGs), to model the local interaction structure in non-cooperative multi-agent sequential decision-making. We define a zero-sum NMG as a model where {the payoffs of the auxiliary games associated with each state are zero-sum and} have some separable (i.e., polymatrix) structure across the neighbors over some interaction network. We first identify the necessary and sufficient conditions under which an MG can be presented as a zero-sum NMG, and show that the set of Markov coarse correlated equilibrium (CCE) collapses to the set of Markov Nash equilibrium (NE) in these games, in that the product of per-state marginalization of the former for all players yields the latter. Furthermore, we show that finding approximate Markov emph{stationary} CCE in infinite-horizon discounted zero-sum NMGs is exttt{PPAD}-hard, unless the underlying network has a ``star topology''. Then, we propose fictitious-play-type dynamics, the classical learning dynamics in normal-form games, for zero-sum NMGs, and establish convergence guarantees to Markov stationary NE under a star-shaped network structure. Finally, in light of the hardness result, we focus on computing a Markov emph{non-stationary} NE and provide finite-iteration guarantees for a series of value-iteration-based algorithms. We also provide numerical experiments to corroborate our theoretical results.
