Q-learners Can Provably Collude in the Iterated Prisoner's Dilemma

2023-12-13

arXiv.org

AI Summary

This work investigates the emergence of cooperative behavior—specifically, implicit collusion—in the repeated prisoner’s dilemma via multi-agent Q-learning. We employ optimistic initialization in Q-learning within a self-play training framework and conduct rigorous fixed-point convergence analysis. Our key contribution is the first formal proof that, under standard assumptions, Q-learners converge stably to the Pareto-optimal Pavlov strategy (“win-stay, lose-shift”), rather than the socially inferior “always-defect” equilibrium commonly presumed in game-theoretic analyses of learning agents. This result challenges the prevailing view that reinforcement learning agents inevitably settle into noncooperative equilibria. Moreover, it establishes a formal, falsifiable mechanism for implicit collusion formation. By characterizing conditions under which cooperation emerges endogenously from decentralized learning dynamics, our analysis provides a foundational theoretical basis for regulating AI-driven market behaviors and designing robust antitrust policies.

Abstract

The deployment of machine learning systems in the market economy has triggered academic and institutional fears over potential tacit collusion between fully automated agents. Multiple recent economics studies have empirically shown the emergence of collusive strategies from agents guided by machine learning algorithms. In this work, we prove that multi-agent Q-learners playing the iterated prisoner's dilemma can learn to collude. The complexity of the cooperative multi-agent setting yields multiple fixed-point policies for $Q$-learning: the main technical contribution of this work is to characterize the convergence towards a specific cooperative policy. More precisely, in the iterated prisoner's dilemma, we show that with optimistic Q-values, any self-play Q-learner can provably learn a cooperative policy called Pavlov, also referred to as win-stay, lose-switch policy, which strongly differs from the vanilla Pareto-dominated always defect policy.

Problem

• Self-play Q-learners collude in prisoner's dilemma
• Conditions for learning cooperative policy analyzed
• Robustness across deep learning algorithms tested

Innovation

• Self-play Q-learners achieve cooperative outcomes
• Theoretical conditions for Pavlov policy learning
• Validation with deep learning algorithm robustness