Equilibrium stability as a driver of cooperation among Q-learners

2026-07-15

AI Summary

This study investigates under what conditions Q-learning agents with a constant exploration rate exhibit cooperation in the time-average sense rather than converging to competitive strategies in the repeated prisoner’s dilemma. By formulating a stochastic dynamical system based on Q-learning and integrating expected dynamics analysis with high-dimensional simulations, this work introduces a novel perspective that links equilibrium stability to the emergence of cooperation in settings with persistent exploration. The authors derive a theoretical criterion characterizing when cooperative behavior dominates in the time average, which effectively predicts the prevalence of non-defecting strategies in ε-greedy Q-learning. This result provides a rigorous theoretical foundation for understanding the spontaneous emergence of cooperation in multi-agent reinforcement learning systems.

Abstract

Algorithmic collusion among pricing algorithms has raised concerns about sustained supra-competitive prices and their implications for social welfare. Existing work has largely focused on the probability that reinforcement-learning algorithms converge to cooperative strategies, typically under the assumption that exploration vanishes over time. Motivated by the observation that algorithms deployed in practice are likely to continue exploring in order to remain adaptive to changing environments, we study learning dynamics under constant exploration. In this setting, the relevant question is no longer whether an algorithm converges to a particular strategy profile, but rather what fraction of time the algorithms spend playing cooperative strategies. Even in the benchmark case of the repeated Prisoner's Dilemma with one-period memory, this yields high-dimensional stochastic learning dynamics, for which a complete analytic treatment is intractable. We show that cooperative strategies can be dominant in this time-averaged sense and derive a boundary predicting when such dominance arises, based on the expected dynamics of the Q-learning process. Extensive simulations show that this boundary is a strong predictor for non-defection-dominated behaviour under epsilon-greedy Q-learning.

Problem

• equilibrium stability
• algorithmic collusion
• Q-learning
• cooperation
• constant exploration

Innovation

• constant exploration
• time-averaged cooperation
• Q-learning dynamics
• equilibrium stability
• algorithmic collusion